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Why  GAO  Did  This  Study 

The  Marine  Corps’  ACV  is  intended  to 
transport  Marines  from  ship  to  shore 
and  provide  armored  protection  on 
land.  It  is  to  potentially  replace  all  or  a 
portion  of  the  decades  old  AAV  fleet, 
and  is  expected  to  eventually  offer 
increased  amphibious  capability  and 
high  water  speed. 

The  National  Defense  Authorization 
Act  for  Fiscal  Year  2014  included  a 
provision  that  GAO  annually  review 
and  report  on  the  ACV  program  until 
2018.  This  report  provides  an  updated 
discussion  of  (1 )  how  the  ACV 
program’s  efforts  compare  to 
acquisition  best  practices  and 
examines  (2)  how  the  increments  of 
ACV  will  achieve  amphibious 
capability. 

To  conduct  this  work,  GAO  reviewed 
program  documentation  and  other 
materials  for  the  ACV  acquisition  and 
Navy  surface  connector  programs. 

GAO  identified  acquisition  and  analysis 
of  alternatives  best  practices  based  on 
its  prior  body  of  work  and  DOD 
guidance.  GAO  also  interviewed 
program  and  agency  officials. 

What  GAO  Recommends 

GAO  is  not  making  recommendations 
in  this  report.  In  commenting  on  a  draft 
of  this  report,  DOD  stated  that  it 
believes  its  efforts  are  aligned  with 
best  practices  and  that  GAO’s  report 
appears  to  underestimate  ACV  1.1’s 
planned  technical  maturity.  GAO  found 
that  some  program  plans  do  not  align 
with  best  practices  and  that  while  some 
aspects  of  the  acquisition  do  suggest 
lower  levels  of  risk,  these  deviations 
could  potentially  increase  program  risk. 
GAO  will  continue  to  monitor  these 
risks  as  the  program  moves  forward. 

View  GAO-16-22.  For  more  information, 
contact  Marie  A.  Mak  at  (202)  512-4841  or 
makm@qao.gov. 


What  GAO  Found 

Most  of  the  current  activities  of  the  U.S.  Marine  Corps’  Amphibious  Combat 
Vehicle  (ACV)  program  have  demonstrated  the  use  of  best  practices,  but  plans 
for  an  accelerated  acquisition  schedule  pose  potential  risks.  As  the  program 
approaches  the  start  of  engineering  and  manufacturing  development,  it  is 
seeking  to  rely  on  mature  technologies  that  have  been  demonstrated  to  work  in 
their  intended  environment  as  well  as  fostering  competition — a  critical  tool  for 
achieving  the  best  return  on  the  government’s  investment.  Further,  GAO 
analyzed  the  ACV  analysis  of  alternatives  that  the  Marine  Corps  produced  for  the 
initial  portion  of  the  ACV  development,  finding  that  overall  it  met  best  practices 
by,  for  example,  ensuring  that  the  analysis  of  alternatives  process  was  impartial. 
Flowever,  the  Marine  Corps  is  pursuing  an  accelerated  program  schedule  that 
presents  some  risks,  including  plans  to  hold  the  preliminary  design  review  after 
the  start  of  development — a  deviation  from  best  practices  which  could  postpone 
the  attainment  of  information  about  whether  the  design  performs  as  expected. 
Moreover,  GAO  believes  that  the  level  of  planned  concurrency — conducting 
development  testing  and  production  at  the  same  time — could  leave  the  program 
at  greater  risk  of  discovering  deficiencies  after  some  systems  have  already  been 
built,  potentially  requiring  costly  modifications.  Agency  officials  stated  that  mature 
technologies  reduce  risk  and  that,  while  some  concurrency  is  planned,  all 
required  testing  will  be  completed  prior  to  the  production  decision.  While  some 
aspects  of  this  acquisition  do  suggest  lower  levels  of  risk,  these  deviations  could 
potentially  increase  program  risk.  GAO  will  continue  to  monitor  this  risk  as  the 
program  moves  forward. 

The  ACV  program  relies  heavily  on  future  plans  to  increase  ACV  amphibious 
capability  gradually,  in  three  planned  increments  known  as  ACV  1.1,  1.2,  and 
2.0,  but  exactly  how  this  capability  will  be  attained  has  not  yet  been  determined. 

•  ACV  1 .1  -  Although  this  increment  is  expected  to  have  some  amphibious 
capability,  according  to  program  documents,  it  is  expected  to  rely  on  surface 
connector  craft — vessels  that  enable  the  transportation  of  military  assets 
from  ship  to  shore.  Marine  Corps  and  U.S.  Navy  officials  regularly  coordinate 
ACV  1.1  plans  to  operate  with  the  surface  connector  fleet  through 
coordination  mechanisms  such  as  the  Surface  Connector  Council. 

•  ACV  1 .2  -  This  increment  is  expected  to  have  greater  amphibious  capability, 
including  the  ability  to  self-deploy  from  ships.  Based  on  demonstrations  from 
related  programs  to  date,  program  officials  believe  it  will  reach  that  capability, 
but  indicated  that  plans  for  1.2  are  expected  to  depend  on  the  success  of 
ACV  1.1  development. 

•  ACV  2.0  -  This  increment  represents  a  future  decision  point  when  the  Marine 
Corps  plans  to  determine  how  to  replace  the  Assault  Amphibious  Vehicle 
(AAV)  fleet.  The  Marine  Corps  is  currently  exploring  technologies  that  may 
enable  high  water  speed — a  significant  increase  from  the  amphibious  goals 
identified  for  ACV  1.1.  Therefore,  how  it  will  achieve  the  amphibious 
capability  envisioned  for  ACV  2.0  is  undetermined. 
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Abbreviations 
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Department  of  Defense 
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October  28,  2015 
Congressional  Committees 

Since  1972,  the  primary  platform  for  transporting  Marines  from  ship  to 
shore  under  hostile  and  hazardous  conditions  has  been  the  Assault 
Amphibious  Vehicle  (AAV).  In  2011,  acquisition  of  a  proposed 
replacement  vehicle — the  United  States  Marine  Corps’  (USMC) 
Expeditionary  Fighting  Vehicle  (EFV) — was  canceled  following  the 
expenditure  of  $3.7  billion  from  fiscal  year  1995  through  201 1  due  to 
concerns  regarding  the  program’s  affordability.  Also  in  2011,  USMC 
subsequently  began  the  acquisition  process  for  the  Amphibious  Combat 
Vehicle  (ACV),  a  potential  replacement  vehicle  for  all  or  a  portion  of  the 
AAV  fleet.  The  ACV  is  intended  to  transport  Marines  from  ship  to  shore 
and  provide  armored  protection  once  on  land.  The  ACV  acquisition 
approach  calls  for  ACV  development  in  three  increments  with  increasing 
amphibious  capability,  ACV  1.1,  1.2  and  2.0. ,  with  ACV  1.1  scheduled  to 
start  development  in  November  2015. 

The  National  Defense  Authorization  Act  for  Fiscal  Year  2014  included  a 
provision  for  us  to  annually  review  and  report  to  the  congressional 
defense  committees  on  the  ACV  program  until  201 8. 1  Previous  reports  in 
2014  and  2015  described  the  efforts  to  initiate  the  ACV  program  and  how 
its  incremental  acquisition  approach  compares  to  acquisition 
management  best  practices.2  This  report  provides  an  updated  discussion 
of  (1)  how  the  ACV  program’s  efforts  compare  to  acquisition  best 
practices,  and  examines  (2)  how  the  increments  of  ACV  are  to  achieve 
amphibious  capability.  This  report  also  includes  updates  to  the  analysis  of 
alternatives  (AOA)  best  practices  identified  in  prior  GAO  work.3 These 


Vub.  L.  No.  113-66,  §  251  (2013).  The  ACV  program  is  relatively  early  in  the  acquisition 
process.  As  a  result,  we  were  unable  to  review  all  of  the  elements  in  the  mandate  since 
the  ACV  program  has  not  yet  progressed  to  those  stages  in  the  acquisition  process. 

2GAO,  Status  of  Efforts  to  Initiate  an  Amphibious  Combat  Vehicle  Program, 
GAO-14-359R,  (Washington,  D.C.:  Apr.  10,  2014)  and  Amphibious  Combat  Vehicle 
Acquisition:  Marine  Corps  Adopts  an  Incremental  Approach,  GAO-1 5-385,  (Washington, 
D.C.:  Apr.  15,  2015). 

3GAO,  DOE  and  NNSA  Project  Management:  Analysis  of  Alternatives  Could  Be  Improved 
by  Incorporating  Best  Practices,  GAO-15-37  (Washington,  D.C.:  Dec.  11, 2014). 
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updated  best  practices  supersede  those  AOA  best  practices  published  in 
prior  reports. 

To  conduct  this  work,  we  reviewed  program  documentation  and  other 
materials  for  the  ACV  1.1  acquisition,  including  the  acquisition  strategy, 
technology  readiness  assessment  and  2014  ACV  AOA,  as  well  as 
program  documentation  from  Navy  surface  connector  programs.4  We 
identified  acquisition  best  practices  based  on  our  extensive  body  of  work 
in  that  area  and  Department  of  Defense  (DOD)  guidance,  and  used  this 
information  to  analyze  the  proposed  ACV  acquisition  approach  and 
acquisition  activities  to  date.  We  updated  and  refined  the  GAO-identified 
AOA  best  practices  by  soliciting  comments  on  those  best  practices  from  a 
group  of  internal  and  external  experts  and  vetting  these  comments  with 
GAO  experts.  The  resulting  changes  include  the  consolidation  of  some 
best  practices,  reducing  the  number  from  24  to  22,  and  the  establishment 
of  four  characteristics  that  identify  a  high-quality,  reliable  AOA  process. 
Appendix  I  contains  additional  information  on  these  best  practices.  We 
also  reviewed  our  previous  work  on  the  ACV  and  EFV  programs.  In 
addition,  we  interviewed  program  and  agency  officials  from  the  USMC’s 
Advanced  Amphibious  Assault  program  office  and  Combat,  Development, 
and  Integration  Division,  the  Office  of  the  Secretary  of  Defense,  Cost 
Assessment  and  Program  Evaluation,  the  Naval  Sea  Systems  Command, 
and  the  Office  of  the  Assistant  Secretary  of  the  Navy  for  Research, 
Development,  and  Acquisition.  Appendix  II  contains  additional  details 
about  our  scope  and  methodology. 

We  conducted  this  performance  audit  from  May  2015  to  October  2015  in 
accordance  with  generally  accepted  government  auditing  standards. 
Those  standards  require  that  we  plan  and  perform  the  audit  to  obtain 
sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives.  We  believe  that 
the  evidence  obtained  provides  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives. 


Background 


The  AAV  is  a  tracked  (non-wheeled)  vehicle  with  the  capability  to  self¬ 
deploy — or  launch  from  ships  (see  figure  1 ).  The  AAV  has  a  water  speed 


4Surface  connector  craft  are  vessels  that  enable  the  transportation  of  military  assets  from 
ship  to  shore.  ACV  1 .1  is  expected  to  rely  on  surface  connector  craft  for  such 
transportation. 
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of  approximately  six  knots,  and  is  usually  deployed  from  within  sight  of 
the  shore,  a  factor  that  poses  survivability  risks  in  certain  threat 
environments.  According  to  USMC  officials,  the  AAV  has  become 
increasingly  difficult  to  maintain  and  sustain.  As  weapons  technology  and 
the  nature  of  threats  have  evolved  over  the  past  four  decades,  the  AAV  is 
viewed  as  having  limitations  in  water  speed,  land  mobility,  lethality, 
protection,  and  network  capability.  According  to  DOD,  the  need  to 
modernize  USMC’s  ability  to  move  personnel  and  equipment  from  ship  to 
shore  is  essential.  In  the  last  15  years,  USMC  has  undertaken  a  number 
of  efforts  to  do  this. 


Figure  1:  United  States  Marine  Corps’  Assault  Amphibious  Vehicle 


Source:  United  States  Manne  Corps.  |  GAO- 16-22 


EFV:  USMC  began  development  of  the  EFV  in  2000. 5  The  EFV  was  to 
travel  at  higher  water  speeds — around  20  knots — which  would  have 
allowed  transporting  ships  to  launch  the  EFV  further  from  shore  than  the 


5The  EFV  entered  the  acquisition  process  in  1995  as  the  Advanced  Amphibious  Assault 
Vehicle. 
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AAVs  it  was  to  replace.  However,  following  a  2007  breach  of  a  statutory 
cost  threshold,  that  program  was  restructured  and  subsequently,  in  2011, 
canceled  by  DOD  due  to  affordability  concerns.6 

ACV:  In  2011,  the  USMC  completed  initial  acquisition  documentation 
providing  the  performance  requirements  for  a  new  replacement 
amphibious  vehicle  called  the  ACV.  The  ACV  was  expected  to  be  self¬ 
deploying  with  a  water  speed  of  8  to  12  knots  which  would  permit 
deployment  beyond  the  visual  range  of  the  shore,  but  would  not  achieve 
high  water  speed.7  It  was  also  expected  to  provide  for  sustained 
operations  on  shore  with  improved  troop  protection.  However,  USMC 
leadership  then  requested  an  affordability  analysis  be  completed  that 
would  explore  the  technical  feasibility  of  integrating  high  water  speed  into 
ACV  development.  According  to  DOD  officials,  the  analysis  indicated  that 
achieving  high  water  speed  was  technically  possible  but  required 
unacceptable  tradeoffs  as  the  program  attempted  to  balance  vehicle 
weight,  capabilities,  and  cost.  Meanwhile,  the  USMC  retained  a 
requirement  to  provide  protected  land  mobility  in  response  to  the  threat  of 
improvised  explosive  devices — a  requirement  the  AAV  could  not  meet 
due  to  its  underbody  design.  In  2014  we  reported  that,  according  to 
program  officials,  the  program  office  was  in  the  process  of  revising  its 
ACV  acquisition  approach  based  on  this  affordability  analysis. 

ACV  1.1,  1.2  and  2.0:  In  2014,  the  USMC  revised  its  ACV  acquisition 
approach,  adopting  a  plan  to  develop  the  ACV  in  three  increments: 

•  The  first  increment  of  ACV  development — ACV  1.1 — is  planned  to  be 
a  wheeled  vehicle  that  would  provide  improved  protected  land  mobility 


6Section  2433  of  title  10  of  the  United  States  Code,  commonly  referred  to  as  Nunn- 
McCurdy,  requires  DOD  to  notify  Congress  whenever  a  major  defense  acquisition 
program’s  unit  cost  experiences  cost  growth  that  exceeds  certain  thresholds.  This  is 
commonly  referred  to  as  a  Nunn-McCurdy  breach.  Significant  breaches  occur  when  the 
program  acquisition  unit  cost  or  procurement  unit  cost  increases  by  at  least  15  percent 
over  the  current  baseline  estimate  or  at  least  30  percent  over  the  original  estimate.  For 
critical  breaches,  when  these  unit  costs  increase  at  least  25  percent  over  the  current 
baseline  estimate  or  at  least  50  percent  over  the  original,  DOD  is  required  to  take 
additional  steps,  including  conducting  an  in-depth  review  of  the  program.  Programs  with 
critical  breaches  must  be  terminated  unless  the  Secretary  of  Defense  certifies  to  certain 
facts  related  to  the  program  and  takes  other  actions,  including  restructuring  the  program. 
10  U.S.C.  §  2433a. 

7The  ACV  program  has  evolved  since  this  time.  The  current  requirements  for  the  ACV  1.1 
include  a  water  speed  of  5  to  8  knots. 
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and  limited  amphibious  capability.  The  ACV  1.1  is  expected  to  be  part 
of  an  amphibious  assault  through  the  use  of  surface  connector  craft  to 
travel  from  ship  to  shore.  Surface  connectors  are  vessels  that  enable 
the  transportation  of  military  assets,  including  personnel,  material,  and 
equipment,  from  a  sea  base  or  ship  to  the  shore.8  ACV  1 .1,  a 
successor  to  the  previously  suspended  Marine  Personnel  Carrier 
program,  is  using  prototypes,  demonstration  testing,  and  other  study 
results  from  that  program.9  DOD  officials  estimated  that,  in  comparing 
the  past  Marine  Personnel  Carrier  program  and  the  ACV  1 .1  as 
currently  envisioned,  the  two  are  about  98  percent  the  same.  Troop 
capacity — nine  for  the  Marine  Personnel  Carrier  and  a  threshold,  or 
minimum,  of  10  for  the  ACV  1.1 — is  the  main  difference  between  the 
two.  Figure  2  provides  a  notional  drawing  of  the  ACV  1.1. 

•  The  second  increment — ACV  1 .2 — adds  two  variants  of  the  vehicle  for 
other  uses  and  aims  to  improve  amphibious  capability.  Program 
officials  anticipate  that  it  will  demonstrate  amphibious  capability  that 
matches  the  AAV,  including  the  ability  to  self-deploy  and  swim  to 
shore.  According  to  DOD  officials,  ACV  1 .2  will  be  based  on  the 
results  of  ACV  1.1  testing  and  it  is  anticipated  that  some  1.1s  will  be 
upgraded  with  ACV  1.2  modifications. 

•  The  third  effort,  referred  to  as  ACV  2.0,  focuses  on  technology 
exploration  to  attain  high  water  speed — a  critical  capability,  according 
to  DOD  officials.  These  technology  exploration  efforts  are  seeking 
design  options  that  may  enable  high  water  speed  capability  without 
accruing  unacceptable  trade-offs  in  other  capabilities,  cost  or 
schedule.  According  to  officials,  ACV  2.0  is  a  conceptual  placeholder 
for  a  future  decision  point  when  the  Marine  Corps  plans  to  determine 


8A  sea  base  is  an  area  that  is  intended  to  “provide  a  sovereign,  maneuverable  and  secure 
area  that  can  be  used  to  assemble,  project  and  sustain  combat  power  relatively 
unconstrained  by  political  and  diplomatic  restrictions.”  United  States  Navy,  Sea  Base 
Branch,  Expeditionary  Warfare  Directorate,  Sea  Base  Branch:  Transforming  Naval 
Expeditionary  Warfare  for  a  New  Strategic  Environment,  accessed  on  September  2,  2015, 
http://www.navy.mil/N85/SB.html. 

9The  Marine  Personnel  Carrier  program,  suspended  in  2013,  was  intended  to  provide 
armor-protected  transportation  of  Marines.  According  to  USMC  officials,  budget 
uncertainty  led  the  USMC  to  determine  that  it  could  not  afford  to  have  three  simultaneous 
development  and  procurement  programs  for  armored  vehicles,  specifically  the  ACV,  the 
Marine  Personnel  Carrier,  and  the  Joint  Light  Tactical  Vehicle.  After  considering  strategic 
priorities,  the  USMC  decided  to  suspend  the  Marine  Personnel  Carrier  program  and 
continue  with  the  ACV  and  Joint  Light  Tactical  Vehicle. 
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how  to  replace  the  AAV  fleet,  which  is  expected  to  occur  in  the  mid- 
2020s.  High  water  speed  capability  may  ultimately  be  achieved 
through  an  amphibious  vehicle  or  a  surface  connector  craft. 


Figure  2:  Notional  Drawing  of  Amphibious  Combat  Vehicle  1 .1 


Source:  United  States  Marine  Corps.  |  GAO- 16-22 


Knowledge-Based  Our  prior  work  on  best  practices  has  found  that  successful  programs  take 

Acquisition  Framework  steps  to  gather  knowledge  that  confirms  that  their  technologies  are 

mature,  their  designs  are  stable,  and  their  production  processes  are  in 
control.10  The  knowledge-based  acquisition  framework  involves  achieving 
the  right  knowledge  at  the  right  time,  enabling  leadership  to  make 
informed  decisions  about  when  and  how  best  to  move  into  various 
acquisition  phases.  Successful  product  developers  ensure  a  high  level  of 
knowledge  is  achieved  at  key  junctures  in  development,  characterized  as 
knowledge  points.  Knowledge  Point  1  falls  early  in  the  acquisition  process 


10For  example,  see  GAO,  Defense  Acquisitions:  Assessments  of  Selected  Weapon 
Programs,  GAO-15-342SP,  (Washington,  D.C.:  Mar.  12,  2015  [Re-issued  Apr.  9,  2015]). 
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and  coincides  with  a  program’s  acquisition’s  decision  to  begin 
development,  referred  to  as  Milestone  B.  At  this  knowledge  point,  best 
practices  are  to  ensure  a  match  between  resources  and  requirements. 
Achieving  a  high  level  of  technology  maturity  and  preliminary  system 
design  backed  by  robust  systems  engineering  is  an  important  indicator  of 
whether  this  match  has  been  made.  This  means  that  the  technologies 
needed  to  meet  essential  product  requirements  have  been  demonstrated 
to  work  in  their  intended  environment.  In  addition,  the  developer  has 
completed  a  preliminary  design  of  the  product  that  shows  the  design  is 
feasible.  Figure  3  identifies  the  ACV  1.1  acquisition’s  status  within  the 
DOD  acquisition  process. 


Figure  3:  Alignment  of  DOD’s  Acquisition  Process  and  Best  Practices 

Department  of  Defense  (DOD)  acquisition  process: 


Current  status  of 
Amphibious  Combat 
Vehicle  (ACV)  1.1 


Milestones: . 


A 

Materiel  solution  analysis 

i 


>  Planned  entry  of 
ACV  1.1  into  DOD's 
acquisition  process 


.Development  start 


.Production  start 


▲  Materiel  development  Preliminary  design  ▲  ( 

~  decision  review  (PDR)  ~ 

k  Critical  design  i 

review  (CDR)  ’ 

4 

►  < 
► 

► 

Best  practices  knowledge-based  acquisition  model: 

-•  Knowledge  Point  1 

Technologies,  time,  funding 
and  other  resources  match 
customer  needs. 

Decisions  to  invest  in  product 
development. 

Key  steps: 

-•  Knowledge  Point  2 

Design  is  stable  and 
performs  as  expected. 

Decisions  to  start  building 
and  testing  production 
representative  prototypes. 

Key  steps: 

-«  Knowledge  Point  3 

Production  meets  cost, 
schedule,  and  quality  target. 

Decisions  to  produce  first 
units  for  customer. 

Key  steps: 

•  PDR  completed 

•  Technologies  demonstrated 
to  high  levels 

•  Incremental  acquisition 
strategy  in  place 

•  Knowledge-based  cost 
estimate 


•  Subsystems  and  system 
level  CDRs  completed 

•  Ninety  percent  of 
engineering  drawings 
released 

•  Eariy  integrated  system 
prototype  demonstrated 

•  Critical  manufacturing 
process  identified 


•  Fully  integrated,  capable 
prototype  demonstrated  in 
intended  environment. 

•  Manufacturing  processes  in 
control 

•  Product  reliability 
demonstrated 


Source:  GAO  analysis  of  DOD-provided  data,  DOD  Instruction  5000.02,  "Operation  of  the  Defense  Acquisition  System"  (January  7, 2015)  and  GAO's  best  practices.  |  GAO-16-22 
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Early  ACV  Acquisition 
Activities 

Demonstrate  Best 
Practices 


Our  review  of  the  available  documents  that  have  been  prepared  to  inform 
the  November  201 5  decision  to  begin  system  development  of  ACV  1.1  — 
including  the  acquisition  strategy  and  an  updated  2014  AOA — found  that 
most  of  the  ACV  program’s  acquisition  activities  to  date  reflect  the  use  of 
best  practices.  The  incremental  approach  to  achieving  full  capability  itself 
is  consistent  with  best  practices.  The  ACV  1.1  acquisition  strategy 
minimizes  program  risk  by  using  mature  technology,  competition,  and 
fixed-price  type-contracts  when  possible.  In  addition,  our  analysis  of  the 
2014  AOA  found  that  overall  it  met  best  practices.  Going  forward, 
however,  some  elements  of  the  acquisition  approach,  for  example,  the 
program’s  plan  to  hold  a  preliminary  design  review  (PDR) — a  technical 
review  assessing  the  system  design — after  beginning  development,  do 
not  align  with  best  practices  and  could  increase  program  risk.  While  some 
aspects  of  this  acquisition  do  suggest  lower  levels  of  risk,  these 
deviations  could  potentially  increase  program  risk.  GAO  will  continue  to 
monitor  this  risk  as  the  program  moves  forward. 


Aspects  of  the  ACV  1 . 1 
Acquisition  Strategy 
Demonstrate  Efforts  to 
Minimize  Risk 


The  ACV  1.1  acquisition  strategy  prepared  to  inform  the  upcoming  start  of 
engineering  and  manufacturing  development  minimizes  program  risk  by 
following  best  practices,  such  as  using  mature  technology,  competition, 
and  fixed-price-type  contracts  when  possible. 

Technology  maturity.  The  ACV  program  plans  to  utilize  mature 
technology  in  ACV  1.1  development.  According  to  acquisition  best 
practices,  demonstrating  a  high  level  of  maturity  before  allowing  new 
technologies  into  product  development  programs  puts  programs  in  a 
better  position  to  succeed.11  To  support  a  decision  to  begin  development, 
a  technology  readiness  assessment  (TRA)  was  performed  to  assess  the 
maturity  of  critical  technologies  to  be  integrated  into  the  program.  DOD 
defines  critical  technology  elements  as  new  or  novel  technology  that  a 
platform  or  system  depends  on  to  achieve  successful  development  or 
production  or  to  successfully  meet  a  system  operational  threshold 
requirement.  In  a  TRA,  identified  critical  technologies  are  assessed 
against  a  technological  readiness  level  (TRL)  scale  of  1  to  9.  Specifically, 
a  rating  of  TRL  1  demonstrates  “basic  principles  observed  and  reported,” 
and  TRL  9  demonstrates  “actual  system  proven  through  successful 


1 1  GAO,  Best  Practices:  Better  Management  of  Technology  Development  Can  Improve 
Weapon  System  Outcomes,  GAO/NSIAD-99-162  (Washington,  D.C.:  July  30,  1999). 
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mission  operations.”12  Overall,  the  completed  ACV  1.1  TRA  assessed  the 
program  at  TRL  7,  indicating  demonstration  in  an  operational 
environment.  This  assessment  was  based  on  the  non-developmental 
nature  of  the  vehicles,13  the  use  of  mature  technology  for  modifications, 
and  tests  and  demonstrations  of  prototype  vehicles  done  under  the 
Marine  Personnel  Carrier  program. 

The  TRA  identified  two  critical  technology  elements.  One  version  of  a 
critical  technology — the  Drivers  Vision  Enhancement — had  not  been 
proven  in  the  marine  environment.  Subsequent  to  the  TRA  report,  the 
version  selected  was  replaced  by  a  system  that  has  been  used  on  the 
AAV  for  10  years  and  is  proven  in  a  marine  environment.  As  a  result,  it  is 
assessed  at  a  TRL  8.  The  second  critical  technology,  the  Remote 
Weapon  Station,  was  assessed  at  TRL  7.  However,  the  operational 
environment  in  which  this  technology  has  been  demonstrated  is  not  the 
same  environment  it  will  face  as  part  of  the  ACV  1.1 .  Specifically,  the 
ACV  1 .1  is  expected  to  operate  in  a  marine  environment  and,  therefore, 
the  weapon  system  must  be  prepared  to  function  and  be  maintained  in 
the  same  environment.  According  to  agency  officials,  some  of  the 
mitigation  steps  that  have  been  identified  to  help  prepare  the  system  for 
the  marine  environment,  specifically,  incorporating  techniques  used  on 
another  version  of  the  system  that  is  fielded  on  a  Navy  patrol  boat,  have 
not  yet  been  applied.  These  factors  add  an  element  of  risk  to  the  use  of 
the  technology  and  suggest  a  lower  assessed  level  of  technology  maturity 
that  is  in  line  with  DOD  policy  and  statute,14  which  generally  require  a 
TRL  6  prior  to  starting  development,  but  is  not  in  line  with  best  practices, 
which  call  for  a  TRL  7. 15  In  the  acquisition  strategy,  the  program  has 


12Department  of  Defense,  Technology  Readiness  Assessment  (TRA)  Deskbook  (July 
2009). 

1  R 

A  non-developmental  item  is  a  previously  developed  item  of  supply  used  exclusively  for 
governmental  purposes  by  a  Federal  agency,  a  State  or  local  government,  or  a  foreign 
government  with  which  the  U.S.  has  a  mutual  defense  cooperation  agreement;  or  such  an 
item  that  only  requires  minor  modifications  or  modifications  of  a  type  customarily  available 
in  the  commercial  marketplace  in  order  to  meet  the  requirements  of  the  procuring  agency; 
or  any  item  of  supply  being  produced  that  doesn’t  meet  this  definition  solely  because  the 
item  is  not  yet  in  use.  Federal  Acquisition  Regulation  (FAR)  §  2.101. 

14 

A  major  defense  acquisition  program  generally  may  not  receive  approval  for  system 
development  start  until  the  milestone  decision  authority  certifies  that  the  technology  in  the 
program  has  been  demonstrated  in  a  relevant  environment.  10  U.S.C.  §  2366b(a)(3)(D). 

^Demonstration  in  a  relevant  environment  is  TRL  6.  Demonstration  in  an  operational 
environment  is  TRL  7. 
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identified  adapting  the  Remote  Weapon  Station  to  the  marine 
environment  as  a  principal  program  risk  because  using  the  system  under 
different  operational  conditions  may  have  a  significant  impact  on  system 
reliability.  While  the  program  has  identified  additional  risk  mitigation 
strategies — including  planned  component  testing  during  development  and 
development  of  preventative  maintenance  procedures — this  technology 
could  entail  a  somewhat  higher  level  of  risk  than  the  TRL  level  suggests 
and  may  require  additional  attention  as  development  begins. 

Competition.  According  to  our  prior  work,  competition  is  a  critical  tool  for 
achieving  the  best  return  on  the  government’s  investment.16  The  ACV 
acquisition  approach  has  fostered  competition  in  the  acquisition  process, 
both  through  competitive  prototyping  that  took  place  prior  to  the  start  of 
development  and  with  competition  that  continues  through  development 
until  production.  Specifically,  before  the  Marine  Personnel  Carrier 
program  was  suspended,  the  government  awarded  a  contract  to  test 
critical  sub-systems  including  the  engine,  transmission,  suspension  and 
hydraulic  hardware  systems.  The  government  also  awarded  four 
contracts  for  system-level  prototypes  demonstrating  the  swim  capability, 
personnel  carry  capability,  and  survivability  of  each  company’s  vehicle. 
The  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics — the  ACV  Milestone  Decision  Authority — has  certified  to  the 
congressional  defense  committees  that  the  ACV  program  had  met  the 
competitive  prototyping  requirement  based  on  the  work  done  under  the 
Marine  Personnel  Carrier  program.  In  addition,  after  development  begins, 
the  program  plans  to  award  ACV  1.1  development  contracts  to  two 
vendors,  maintaining  competition  until  they  select  one  vendor  at  the  start 
of  production. 

Contract  strategy.  When  development  begins,  the  ACV  program  plans 
to  award  hybrid  contracts  to  each  of  the  to-be-selected  developers. 
According  to  program  plans,  each  contract  is  to  utilize  three  different 
pricing  structures  for  different  activities:  fixed-price-incentive  for  ACV  1.1 


16GAO-15-342SP. 


Page  10 


GAO-16-22  Amphibious  Combat  Vehicle 


vehicle  development,17  firm-fixed-price  for  the  delivery  incentive  to  deliver 
test  vehicles  early,18  and  cost-plus-fixed-fee19  for  test  support  and 
advanced  capability  improvements  and  studies.  According  to  the  Federal 
Acquisition  Regulation,  it  is  usually  to  the  Government’s  advantage  for  the 
contractor  to  assume  substantial  cost  responsibility  and  an  appropriate 
share  of  the  cost  risk;  therefore,  fixed-price  incentive  contracts  are 
preferred  when  contract  costs  and  performance  requirements  are 
reasonably  certain.20  Manufacturing  the  development  vehicles  is  the 
largest  anticipated  portion  of  ACV  development  contract  costs.  According 
to  the  ACV  1.1  acquisition  strategy,  a  fixed-price-incentive  contract  is 
considered  the  most  appropriate  contract  type  to  utilize  for  the  vehicle’s 
development  because  the  vehicles  themselves  are  non-developmental  in 
nature  but  there  is  some  risk  related  to  the  integration  of  selected 
systems,  such  as  the  Remote  Weapon  Station,  and  other  modifications 
required  to  meet  USMC  requirements.  Meanwhile,  the  strategy  states  that 
the  delivery  incentive  is  to  be  a  firm-fixed-price,  as  the  fee  is  a  set  dollar 
amount  based  on  how  early  the  vehicles  are  delivered  and  is  not  subject 
to  adjustment  based  on  the  vendor’s  costs. 

Under  cost-reimbursement  contract  types,  such  as  a  cost-plus-fixed-fee 
contract,  the  government  bears  the  risk  of  increases  in  the  cost  of 
performance.  Cost-reimbursement  contract  types  are  suitable  when 


17DOD  plans  for  contractors  to  be  incentivized  to  control  vehicle  development  costs 
through  the  use  of  the  fixed-price  incentive  pricing  structure.  A  fixed-price  incentive 
contract  is  a  fixed-price  contract  that  provides  for  adjusting  profit  and  establishing  the  final 
contract  price  by  application  of  a  formula  based  on  the  relationship  of  total  final  negotiated 
cost  to  total  target  cost.  The  final  price  is  subject  to  a  price  ceiling,  negotiated  at  the 
outset.  The  ACV  development  contract  is  expected  to  share  cost  savings  and  cost 
overruns  with  the  contractor  using  a  50/50  split  and  provide  a  cost  ceiling  of  120  percent 
of  the  target  cost.  FAR  §  16.403. 

1  ft 

A  firm-fixed  price  delivery  incentive  fee  is  also  to  be  offered.  The  firm-fixed-price  is  not 
subject  to  any  adjustment  on  the  basis  of  the  contractor’s  cost  experience  in  performing 
the  contract.  The  firm-fixed  price  delivery  incentive  fee  is  a  set  dollar  amount  that  is  to  be 
determined  based  on  how  early  the  contractors  deliver  the  prototype  vehicles  to  the 
testing  location.  FAR  §  16.202-1. 

19For  the  cost-plus-fixed-fee  activities,  the  government  pays  for  allowable  incurred  costs  to 
the  extent  prescribed  in  the  contract,  as  well  as  a  negotiated  fee  that  is  fixed  at  the  outset. 
The  contract  establishes  an  estimate  of  total  cost  for  the  purpose  of  obligating  funds  and 
establishing  a  ceiling  that  the  contractor  may  not  exceed  without  the  approval  of  the 
contracting  officer.  FAR  §  16.301-1,  16.306(a). 

20FAR§  16.401(c). 
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uncertainties  in  requirements  or  contract  performance  do  not  permit  the 
use  of  fixed-price  contract  types.  A  cost-plus-fixed-fee  structure  is 
planned  for  test  support  before  and  after  the  start  of  production,  vehicle 
transportation  and  other  test-related  activities.  According  to  program 
officials,  the  scope  and  nature  of  these  activities  are  difficult  to  predict, 
making  the  cost-plus-fixed-fee  structure  appropriate.  Officials  also  stated 
that  the  cost-plus-fixed-fee  activities  are  expected  to  comprise  about  1 1 
percent  of  the  total  contract  value. 

Requirements  and  cost  estimates.  Additional  key  documents  have 
been  prepared,  or  are  underway,  in  accordance  with  DOD  policy.  The 
ACV  1.1  Capabilities  Development  Document,  providing  the  set  of 
requirements  for  development,  is  tailored  specifically  for  ACV  1.1.  In 
accordance  with  DOD  policy,  the  ACV  1.1  Capabilities  Development 
Document  was  validated  prior  to  the  release  of  the  ACV  1.1  request  for 
proposal  in  March  2015.  In  addition,  best  practices  and  DOD  policy  also 
call  for  the  development  of  an  independent  cost  estimate  prior  to  the  start 
of  development.  According  to  agency  officials,  the  independent  cost 
estimate  is  underway  and  will  be  prepared  for  the  Milestone  B  decision. 
The  acquisition  strategy  identifies  no  funding  shortfalls  for  the  program  as 
of  the  fiscal  year  2016  President’s  budget  submission. 


2014  ACV  Analysis  of 
Alternatives  Met  Best 
Practices 


Our  assessment  of  the  2014  AOA  found  that  overall  it  met  best  practices 
for  AOAs  and  is,  therefore,  considered  reliable.  An  AOA  is  a  key  first  step 
in  the  acquisition  process  intended  to  assess  alternative  solutions  for 
addressing  a  validated  need.  AOAs  are  done  or  updated  to  support  key 
acquisition  decision  points.  The  USMC  completed  an  AOA  update  for 
ACV  1.1  in  late  2014  to  support  the  release  of  the  ACV  1.1  request  for 
proposal.  Over  the  years,  other  AOAs  have  been  completed  for  related 
acquisitions,  including  the  EFV,  the  Marine  Personnel  Carrier  and  the 
previous  version  of  the  ACV  considered  in  2012.  These  previous  AOAs 
and  other  supporting  studies  comprise  a  body  of  work  that  has  informed 
the  most  recent  ACV  AOA  update  as  well  as  the  ACV  1.1  acquisition  as  a 
whole. 

AOAs  can  vary  in  quality,  which  can  affect  how  they  help  position  a 
program  for  success.  We  have  previously  identified  best  practices  for  the 
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development  of  AOAs.21  Considered  in  the  context  of  the  related  AOA 
body  of  work,  the  ACV  AOA  met  1 5  of  the  22  AOA  best  practices, 
including  ensuring  that  the  AOA  process  was  impartial  and  developing  an 
AOA  process  plan,  among  others.  Further,  four  of  the  remaining  best 
practices  were  substantially  met,  two  were  partially  met,  and  one  was 
minimally  met.22  For  example,  best  practices  call  for  the  documentation  of 
all  assumptions  and  constraints  used  in  the  analysis.  We  found  that  the 
2014  AOA  does  not  include  a  full  list  of  assumptions  and  constraints  and 
any  assumptions  or  constraints  from  previous  analysis,  if  relevant,  were 
not  updated  or  referenced  in  the  new  analysis.  As  a  result,  it  could  be 
difficult  for  decision  makers  to  make  comparisons  and  trade-offs  between 
alternatives.  Appendices  1  and  II  provide  more  information  on  the 
methodology  used  in  this  analysis  and  appendix  III  provides  the  results  of 
our  AOA  analysis  in  greater  detail,  DOD’s  Cost  Assessment  and  Program 
Evaluation  staff  also  reviewed  the  2014  AOA  and  found  that  it  was 
sufficient.  However,  they  identified  a  few  areas  of  caution,  including 
recommending  additional  testing  of  land  mobility  to  further  verify  USMC 
assertions  that  the  wheeled  ACV  1.1  would  have  the  same  mobility  in  soft 
soil  as  tracked  vehicles. 

Accelerated  Schedule 
Presents  Program  Risks 

According  to  USMC  officials,  the  ACV  program  is  pursuing  an  aggressive 
schedule  in  order  to  obtain  ACV  1.1  initial  operational  capability  in  fiscal 
year  2020.  The  program  is  scheduled  to  hold  its  PDR  after  development 
starts,  a  deviation  from  best  practices.  In  addition,  according  to  program 
officials,  as  a  result  of  the  aggressive  acquisition  schedule,  the  program 
plans  on  a  higher  level  of  concurrency  between  development  testing  and 
production  than  would  take  place  under  a  more  typical  acquisition 
schedule.23  This  aggressive  schedule  may  likely  have  congressional 
decision  makers  approve  funds  to  begin  production  based  on  little  to  no 
evidence  from  the  testing  of  delivered  ACV  1.1  prototypes.  Some  factors 
may  mitigate  the  risk  posed  by  this  acceleration,  for  example,  program 
officials  have  stated  that  all  required  testing  will  take  place  prior  to  the 

21GAO-15-37.  This  report  provides  an  update  to  the  best  practices  identified  in 

GAO-15-37.  See  appendix  1. 

22See  appendix  II  for  assessment  level  definitions. 

23Concurrency  is  broadly  defined  as  the  overlap  between  technology  development  and 
product  development  or  between  product  development  and  production. 
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start  of  production.  However,  further  attention  may  be  warranted  in  our 
future  reviews  of  the  program’s  schedule. 

The  ACV  1.1  program  is  planning  to  hold  its  PDR  about  90  days  after 
development  begins  and  to  combine  its  PDR  and  the  critical  design 
review  (CDR)  into  one  event.  Best  practices  recommend  that  the  PDR  is 
held  before  development  begins  in  order  to  increase  the  knowledge 
available  to  the  agency  when  development  starts,  for  example,  increasing 
confidence  that  the  design  will  meet  the  requirements  established  in  the 
Capabilities  Development  Document.  The  absence  of  a  PDR  introduces 
some  risk  by  postponing  the  attainment  of  knowledge  until  after 
development  begins  and  reducing  scheduled  time  to  address  any  design 
issues  that  may  arise.  In  addition,  it  is  a  best  practice  to  demonstrate 
design  stability  at  the  system-level  CDR,  completing  at  least  90  percent  of 
engineering  drawings  at  that  time.  Combining  the  PDR  and  CDR  may 
limit  the  time  available  to  the  program  to  address  any  issues  identified 
and  ensure  that  sufficient  knowledge  is  attained  prior  to  the  program 
moving  forward.  For  example,  in  a  2006  report,  we  found  that  the  EFV 
program’s  CDR  was  held  almost  immediately  after  the  start  of 
development — similar  to  the  approach  for  ACV  1.1 — and  before  the 
system  integration  work  had  been  completed.24  Testing  of  the  early 
prototypes  continued  for  three  years  into  system  development,  well  after 
the  tests  could  inform  the  CDR  decision.  Best  practices  call  for  system 
integration  work  to  be  conducted  before  the  CDR  is  held.  According  to 
DOD  officials,  the  ACV  1.1  PDR  will  be  held  after  Milestone  B  because 
contracts  are  not  planned  to  be  awarded  prior  to  that  time.  In  addition, 
DOD  officials  stated  that  the  technological  maturity  of  ACV  1.1  reduces 
risk  and  permits  both  the  waiver  of  the  PDR  requirement  and  the 
consolidation  of  the  reviews.  While  the  use  of  mature  technology  could 
suggest  a  reduced  risk  from  this  deferral,  we  believe  that  contracts  could 
have  been  awarded  earlier  in  the  acquisition  process  in  order  to  facilitate 
a  PDR  prior  to  development  start. 

The  current  ACV  1.1  program  schedule  demonstrates  concurrency 
between  testing  and  production  that  could  represent  increased  program 
risk.  According  to  agency  officials,  approximately  one  year  of 
development  testing  will  take  place  prior  to  the  program’s  production 


24GAO,  Defense  Acquisitions:  The  Expeditionary  Fighting  Vehicle  Encountered  Difficulties 
in  Design  Demonstration  and  Faces  Future  Risks,  GAO-06-349  (Washington,  D.C.:  May 
1,2006). 


Page  14 


GAO-16-22  Amphibious  Combat  Vehicle 


decision  in  order  to  assess  production  readiness.  Another  ten  months  of 
testing  will  continue  after  the  start  of  production.  The  intent  of 
developmental  testing  is  to  demonstrate  the  maturity  of  a  design  and  to 
discover  and  fix  design  and  performance  problems  before  a  system 
enters  production.  According  to  agency  officials,  the  adoption  of  an 
accelerated  fielding  schedule  is  behind  the  level  of  overlap  between 
developmental  testing  and  production.  They  stated  that  they  plan  to  have 
completed  all  development  testing  and  operational  assessment  required 
to  support  the  production  decision  by  the  time  that  decision  is  made.  DOD 
policy  allows  some  degree  of  concurrency  between  initial  production  and 
developmental  testing  and,  according  to  our  prior  work,  some 
concurrency  may  be  necessary  when  rapidly  fielding  urgently  needed 
warfighter  capabilities.  However,  our  past  work  has  also  shown  that 
beginning  production  before  demonstrating  that  a  design  is  mature  and 
that  a  system  will  work  as  intended  increases  the  risk  of  discovering 
deficiencies  during  production  that  could  require  substantial  design 
changes  and  costly  modifications  to  systems  already  built.  A  detailed  test 
plan  will  not  become  available  until  Milestone  B  as  is  expected  for 
acquisition  programs.  When  such  a  plan  is  available,  we  will  further 
assess  the  risk  presented  by  this  approach. 

Moreover,  under  the  current  ACV  1.1  program  schedule,  Congress  may 
likely  be  called  upon  to  provide  production  funding  for  ACV  1.1  production 
based  on  little  to  no  evidence  from  the  testing  of  delivered  ACV  1.1 
prototypes.  The  program  is  scheduled  to  make  a  production  decision,  and 
select  one  vendor,  in  fiscal  year  201 8.  Under  the  normal  budget  process, 
Congress  would  be  provided  the  request  for  funding  that  production  with 
the  President’s  budget  in  February  2017,  around  the  same  time  that  the 
prototype  ACV  1.1  vehicles  are  scheduled  to  be  delivered.  In  the  event 
that  the  development  testing  schedule  experiences  delays  and  key  tests 
are  postponed  until  after  the  planned  production  decision,  the  program 
may  face  increased  risk. 


Incremental 
Development  of 
Future  Amphibious 
Capability  Is  Key  to 
Current  ACV  Strategy 


The  success  of  the  ACV  acquisition  strategy  depends  upon  the 
attainment  of  improved  amphibious  capabilities  over  time.  The  first 
increment,  ACV  1.1,  is  not  expected  to  have  ship  to  shore  amphibious 
capability  and  thus  is  planned  to  use  Navy  surface  connectors  to  travel 
from  ship  to  shore.  The  USMC  and  the  Navy  have  coordinated  the 
planned  operation  of  ACV  1.1  with  surface  connectors  to  ensure 
compatibility  and  availability.  The  ACV  acquisition  intends  to  rely  heavily 
upon  realizing  a  fully  amphibious  ACV  1.2,  providing  AAV-equivalent 
water  mobility  and  the  ability  to  self-deploy.  However,  the  exact  nature  of 
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ACV  1 .2  and  2.0  is  unknown  at  this  time.  Achieving  the  planned 
capabilities  of  future  ACV  increments  is  highly  dependent  upon  ACV  1.1 
attaining  its  planned  amphibious  capability. 


ACV  1.1  Expects  to  Rely 
on  Surface  Connector 
Craft  for  All  Ship  to  Shore 
Transportation 


While  ACV  1 .1  is  expected  to  have  shore  to  shore  amphibious  capability, 
which  would  enable  the  vehicle  to  cross  rivers  and  inland  waterways,  the 
vehicle  is  also  expected  to  rely  on  Navy  surface  connector  craft  for  ship  to 
shore  transportation.  Connectors  have  become  increasingly  important  as 
USMC  vehicles  have  grown  in  weight.  According  to  USMC  analysis, 
about  86  percent  of  USMC  expeditionary  force  assets  are  too  heavy  or 
over-sized  for  air  transport,  and  need  to  be  transported  by  surface 
connectors. 

The  ACV  1.1  requirements  include  transportability  by  currently  available 
and  planned  Navy  surface  connectors.  Because  several  surface 
connectors  can  transport  the  ACV  1.1,  the  selection  of  specific  surface 
connectors  is  planned  to  be  based  on  an  evaluation  of  mission  needs  and 
connector  capabilities.  Some  current  and  planned  Navy  surface 
connectors  that  could  transport  ACV  1.1  are  described  below.  Appendix 
IV  provides  additional  information  on  the  key  capabilities  of  these 
connectors. 

Landing  Craft  Air  Cushion  (LCAC).  The  LCAC  is  a  high  speed 
hovercraft  that  supports  rapid  movement  from  ship  to  shore,  such  as 
during  an  amphibious  assault.  The  LCAC  is  one  of  the  primary 
connectors  that  provide  ship  to  shore  transportation  of  equipment, 
personnel,  and  vehicles.  The  LCAC,  which  can  access  about  70  percent 
of  the  world’s  beaches,  is  optimized  towards  major  combat  operations 
and  forcible  entry.  The  Navy  currently  has  a  fleet  of  72  LCACs  which 
have  received  upgrades  as  a  result  of  a  service  life  extension  program 
effort.  The  Navy  also  plans  to  provide  additional  LCAC  maintenance  until 
replacement  craft  are  acquired. 


Ship  to  Shore  Connector  (SSC).  The  Navy  plans  to  replace  each  LCAC 
with  an  SSC.  The  SSC,  similar  in  design  to  the  LCAC,  is  planned  to 
maintain  or  improve  upon  LCAC  capabilities  with  an  increased  payload 
capacity,  a  longer  service  life,  and  the  ability  to  operate  in  more  harsh 
marine  environments.  SSC  is  planned  to  reach  initial  operational 
capability  of  6  craft  in  2020  and  full  operational  capability  in  2027. 


Landing  Craft  Utility  (LCU).  The  LCU  is  a  utility  connector  that  supports 
ship  to  shore  movement  in  amphibious  assaults  and  also  participates  in  a 
variety  of  other  missions.  The  LCU  has  a  large  range  and  payload 
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capacity,  but  operates  at  a  slower  speed  compared  to  the  LCAC. 
According  to  Navy  officials,  the  LCU  can  access  about  17  percent  of  the 
world’s  beaches,  and  stops  at  the  waters’  edge  in  order  to  unload  its 
cargo. 

Surface  Connector  (X)  Replacement  (SC(X)R).  According  to  Navy 
officials,  the  aging  LCU  craft  are  planned  to  be  replaced  by  SC(X)R  craft 
in  order  to  maintain  a  total  of  32  LCUs  and  SC(X)Rs.  According  to  the 
Surface  Connector  Council,  the  SC(X)R  is  likely  to  be  larger  and  show 
improvements  in  materials,  propulsion,  maintainability,  and  habitability. 
Production  for  the  SC(X)R  is  planned  to  begin  in  2018. 

Expeditionary  Fast  Transport  (EPF).  The  EPF,  formerly  known  as  the 
Joint  High  Speed  Vessel,  is  a  commercial-based  catamaran  that  provides 
heavy-lift,  high-speed  sealift  mobility.  The  EPF  uses  a  ramp  system  to 
allow  vehicles  to  off-load  at  shipping  ports  or  where  developed 
infrastructure  is  unavailable  (referred  to  as  austere  ports).  The  EPF  is 
planned  to  reach  full  operational  capability  in  the  year  2019. 

Figure  4  illustrates  three  examples  of  how  various  surface  connectors 
could  be  used  to  transport  ACV  1.1  from  ship  to  shore.  For  example, 
ACVs  could  be  loaded  onto  an  Expeditionary  Transfer  Dock  (ESD)  and 
then  on  to  LCACs  or  SSCs  while  the  ESD  maneuvers  towards  the  shore. 
The  LCACs  or  SSCs  would  then  launch  from  the  ESD  and  transport  the 
ACVs  to  shore.  The  ACV  could  also  be  off-loaded  at  an  advanced  base 
— such  as  an  island  located  within  the  operational  area — and  then  loaded 
onto  a  EPF  for  transport  to  a  developed  or  austere  port.  Finally,  the  ACVs 
could  be  directly  loaded  from  ships  on  to  a  LCU  or  SC(X)R  and  taken  to 
shore.  This  graphic  includes  selected  examples  only,  and  does  not 
represent  all  possible  transportation  options. 
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Figure  4:  Examples  of  ACV  1.1  Ship  to  Shore  Transportation 
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Source:  GAO  presentation  of  U  S.  Marine  Corps  and  U  S.  Navy  information.  |  GAO-16-22 


Navy  Officials  Have  Taken 
Actions  to  Mitigate 
Potential  SSC  Risks 


SSC  acquisition  risks  may  have  consequences  for  employment  of  ACV 
1.1.  The  Navy  has  identified  that  it  requires  a  combined  fleet  of  at  least  72 
operational  LCACs  and  SSCs  to  support  ship  to  shore  transportation 
demands.  However,  the  Navy  previously  anticipated  a  lack  of  available 
connectors  from  the  year  2015  through  2024,  with  a  maximum  ‘gap,’  or 
shortage,  of  15  craft  in  2019.  Navy  officials  said  that  this  ‘connector  gap’ 
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has  been  mitigated  with  the  extension  of  the  LCAC  service  life  extension 
program25  and  acceleration  of  the  SSC  acquisition. 

In  a  previous  assessment  of  the  SSC  program,26  we  found  that  the  Navy 
recognizes  three  SSC  technologies  as  potential  risk  areas,  for  which  the 
Navy  recommended  further  testing.  According  to  officials,  since  that 
report,  the  Navy  has  completed  additional  testing  for  software,  drivetrain 
components,  and  engine  endurance  to  further  develop  and  reduce  the 
risk  of  these  technologies.  Navy  officials  said  the  SSC  program  plans  to 
continue  testing  these  technologies  and  remains  on-schedule.  However, 
the  SSC  program  entered  production  in  2015,  more  than  2  years  before 
the  estimated  delivery  of  the  test  vehicle.  This  concurrency  of 
development  and  production  creates  a  potential  risk  of  schedule  overruns 
if  deficiencies  in  the  design  are  not  discovered  until  late  in  testing  and 
retrofits  are  required  for  previously  produced  craft.  Navy  officials  said  that 
the  LCAC  service  life  could  be  further  extended  with  additional 
sustainment  funding  in  the  event  of  SSC  acquisition  delays. 


USMC  and  Navy 
Coordinated  ACV  and 
Surface  Connector  Plans 
with  Various  Mechanisms 


The  USMC  and  Navy  regularly  coordinate  on  the  ACV  1.1  to  facilitate  the 
future  use  of  the  surface  connector  fleet  through  the  Joint  Capabilities 
Integration  Development  System  (JCIDS),  the  Surface  Connector 
Council,  and  other  communication. 

JCIDS.  The  JCIDS  process  is  a  DOD-wide  process  to  identify  and  assess 
capability  needs  and  their  associated  performance  criteria.  The 
Capabilities  Development  Document  for  the  ACV  1.1  was  developed  as 
part  of  the  JCIDS  process.  The  document,  among  other  things,  identified 
key  systems  attributes,  key  performance  parameters,  and  design 
requirements  for  the  ACV  1.1  with  input  from  the  USMC,  the  Navy,  and 
others.  For  example,  it  included  design  requirements  that  allow  the  SSC 
to  transport  two  ACVs,  and  ensure  that  ACVs  can  be  transported  by  other 
connector  craft  as  well. 


25The  LCAC  service  life  extension  program  began  in  2005,  and  included  a  number  of  craft 
improvements  that  were  expected  to  add  about  10  years  of  service  life  to  the  fleet.  The 
post-service  life  extension  program  will  extend  the  use  of  19  LCAC  an  additional  5  to  7 
years.  The  post-service  life  extension  program  began  in  fiscal  year  2015  and  is  planned  to 
run  through  fiscal  year  2020. 

26GAO-1 5-342SP. 
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Surface  Connector  Council  and  working  group.  The  Surface 
Connector  Council  serves  as  a  mechanism  through  which  the  USMC  and 
Navy  coordinate  activities  related  to  surface  connectors  that  are  used  for 
amphibious  shipping.  The  council  has  two  co-chairs:  the  Director  of  the 
Navy’s  Expeditionary  Warfare  Division  and  the  Director  of  the  USMC 
Capabilities  Development  Directorate  who  is  also  the  Deputy 
Commandant  for  Combat  Development  and  Integration.  The  council 
membership  is  drawn  from  several  offices  from  both  the  Navy  and  the 
USMC.  The  Council  is  required  to  meet  at  least  biannually  but,  according 
to  Navy  officials,  in  practice  the  Council  generally  meets  quarterly.  At 
these  meetings,  the  Council  has  previously  discussed  ACV  program  risks, 
such  as  connector  availability  and  the  scarcity  of  space  on  connectors, 
and  associated  risk  mitigation  strategies,  according  to  Navy  officials.  The 
Surface  Connector  Council  also  has  a  working  level  forum,  known  as  the 
Surface  Connector  Roundtable,  which  meets  on  a  monthly  basis 
according  to  Navy  officials. 

Informal  discussions.  In  addition  to  coordination  through  JCIDS  and  the 
Surface  Connector  Council,  officials  said  that  informal  discussions 
between  USMC  and  Navy  officials  occur  frequently  to  coordinate  the  ACV 
and  connector  programs. 


Future  Amphibious 
Capability  to  be 
Determined 


The  exact  nature  of  the  ACV’s  future  amphibious  capability  is  not  yet 
known.  USMC  officials  are  confident  that  the  ACV  1.1  would  not  only 
meet  its  minimum  requirements  for  shore  to  shore  swim  capability,  but 
may  exceed  those  requirements  and  be  able  to  swim  from  ship  to  shore. 
Based  on  tests  and  demonstrations  to  date,  program  officials  also 
expressed  confidence  that  ACV  1.2  will  build  on  the  ACV  1.1  capabilities 
and  have  the  ability  to  self-deploy  from  ships.  However,  according  to 
DOD  officials,  the  capabilities  of  the  ACV  1 .2  are  dependent  upon  the 
success  of  ACV  1 . 1  development.  If  the  ACV  1 . 1  does  not  demonstrate 
the  expected  amphibious  capabilities,  then  more  development  than 
currently  anticipated  may  be  required  for  ACV  1.2  to  achieve  ship  to 
shore  amphibious  capability  and  greater  effort  may  be  needed  to  retro-fit 
ACV  1.1  vehicles  to  achieve  the  same  capabilities.  However,  if  ACV  1.1 
demonstrates  greater  than  expected  amphibious  capability,  then  the 
progression  towards  achieving  the  plans  for  the  ACV  1 .2  may  be  easier. 
Program  documentation  and  analysis  to  date  have  been  done  to  develop 
the  ACV  1.1  strategy  and  plans  and  to  support  ACV  1.1  decisions. 
According  to  DOD  officials,  the  USMC  has  not  yet  determined  whether 
the  development  of  ACV  1.2  will  be  done  through  improvements  within 
the  same  program  or  as  a  separate  program  from  ACV  1.1.  DOD  officials 
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stated  that  the  development  of  ACV  1 . 1  and  1 .2  amphibious  capabilities  is 
also  expected  to  impact  the  nature  of  ACV  2.0.  According  to  DOD 
officials,  with  the  ACV  2.0  decision,  the  ACV  program  expects  to  achieve 
high  water  speed,  a  long-standing  goal  and  a  significant  increase  from  the 
current  amphibious  goals  identified  for  ACV  1.1. 

The  current  USMC  amphibious  strategy  plans  for  an  evolving  mix  of 
ACVs  and  upgraded  and  legacy  AAVs  that  are  to  maintain  the  needed 
combination  of  capabilities  at  any  one  time.  According  to  USMC  officials, 
over  time,  the  ACV  program  plans  to  replace  portions  of  the  AAV  fleet 
with  ACV  increments  as  they  become  available.  This  USMC  strategy,  and 
the  analysis  that  supports  it,  is  based  on  the  assumption  that  ACV  1.2  will 
reach  a  desired  level  of  amphibious  capability  and  that  ACV  1.1  vehicles 
can  be  upgraded  to  that  level.  If,  however,  those  or  other  key  capabilities 
cannot  be  achieved,  revisiting  the  USMC’s  strategy  prior  to  making 
production  decisions  for  ACV  1.1,  particularly  addressing  changes  to  its 
overall  amphibious  strategy  and  potentially  updating  its  analysis  of 
alternatives,  will  be  important.  In  addition,  when  and  how  the  USMC  will 
achieve  the  amphibious  capability  envisioned  for  ACV  2.0  remains  to  be 
determined,  according  to  DOD  officials.  We  will  continue  to  monitor  these 
issues  along  with  the  program’s  performance  against  best  practices  as  it 
progresses  toward  the  Milestone  C  production  decision  currently  planned 
for  the  second  quarter  of  fiscal  year  201 8. 


Agency  Comments 
and  Our  Evaluation 


We  are  not  making  any  recommendations  in  this  report.  DOD  provided 
written  comments  on  a  draft  of  this  report.  The  comments  are  reprinted  in 
appendix  V. 

In  commenting  on  a  draft  of  this  report,  DOD  stated  that  it  believes  its 
efforts  on  this  program  are  aligned  with  our  best  practices  and  that  our 
report  appears  to  underestimate  ACV  1.1’s  planned  technical  maturity 
and  associated  risks.  DOD  stated  that  the  vehicle  is  beyond  the  traditional 
PDR  and  CDR  level  of  maturity  and  conducting  a  combined  PDR  and 
CDR  is  appropriate  for  the  level  of  risk  identified  by  the  Program 
Manager.  As  we  stated  in  this  report,  the  program’s  plan  to  hold  a  PDR 
after  beginning  development  does  not  align  with  best  practices  and 
combining  the  PDR  and  CDR  may  limit  the  time  available  to  the  program 
to  address  any  issues  identified  and  ensure  that  sufficient  knowledge  is 
attained  prior  to  the  program  moving  forward.  Further,  as  we  stated 
earlier,  while  some  aspects  of  this  acquisition  do  suggest  lower  levels  of 
risk,  these  deviations  could  potentially  increase  program  risk — risks  that 
we  will  continue  to  monitor  as  the  program  moves  forward. 
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DOD  also  provided  technical  comments  that  were  incorporated,  where 
appropriate. 


We  are  sending  copies  of  this  report  to  interested  congressional 
committees;  the  Secretary  of  Defense;  the  Under  Secretary  of  Defense 
for  Acquisition,  Technology,  and  Logistics;  the  Secretary  of  the  Navy;  and 
the  Commandant  of  the  Marine  Corps.  This  report  also  is  available  at  no 
charge  on  GAO’s  website  at  http://www.gao.gov. 

Should  you  or  your  staff  have  any  questions  on  the  matters  covered  in 
this  report,  please  contact  me  at  (202)  512-4841  or  makm@gao.gov. 
Contact  points  for  our  Offices  of  Congressional  Relations  and  Public 
Affairs  may  be  found  on  the  last  page  of  this  report.  Key  contributors  to 
this  report  are  listed  in  appendix  VI. 


Marie  A.  Mak 

Director,  Acquisition  and  Sourcing  Management 
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List  of  Committees 


The  Honorable  John  McCain 
Chairman 

The  Honorable  Jack  Reed 
Ranking  Member 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Thad  Cochran 
Chairman 

The  Honorable  Richard  J.  Durbin 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
United  States  Senate 

The  Honorable  Mac  Thornberry 
Chairman 

The  Honorable  Adam  Smith 
Ranking  Member 
Committee  on  Armed  Services 
House  of  Representatives 

The  Honorable  Rodney  Frelinghuysen 
Chairman 

The  Honorable  Pete  Visclosky 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
House  of  Representatives 
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Background  and 
Introduction 


Many  guides  have  described  an  approach  to  analyses  of  alternatives 
(AOAs);  however,  there  is  no  single  set  of  practices  for  the  AOA  process 
that  has  been  broadly  recognized  by  both  the  government  and  private- 
sector  entities.  GAO  has  identified  22  best  practices  for  an  AOA  process 
by  (1)  compiling  and  reviewing  commonly  mentioned  AOA  policies  and 
guidance  used  by  different  government  and  private-sector  entities  and  (2) 
incorporating  experts’  comments  on  a  draft  set  of  practices  to  develop  a 
final  set  of  practices.1 

These  practices  can  be  applied  to  a  wide  range  of  activities  in  which  an 
alternative  must  be  selected  from  a  set  of  possible  options,  as  well  as  to  a 
broad  range  of  capability  areas,  projects,  and  programs.  These  practices 
can  provide  a  framework  to  help  ensure  that  entities  consistently  and 
reliably  select  the  project  alternative  that  best  meets  mission  needs.  The 
guidance  below  is  meant  as  an  overview  of  the  key  principles  that  lead  to 
a  successful  AOA  process  and  not  as  a  “how  to”  guide  with  detailed 
instructions  for  each  best  practice  identified. 

The  22  best  practices  that  GAO  identified  are  grouped  into  the  following 
five  phases: 

1 .  Initialize  the  AOA  process:  includes  best  practices  that  are  applied 
before  starting  the  process  of  identifying,  analyzing,  and  selecting 
alternatives.  This  includes  determining  the  mission  need  and 
functional  requirements,  developing  the  study  time  frame,  creating  a 
study  plan,  and  determining  who  conducts  the  analysis. 

2.  Identify  alternatives:  includes  best  practices  that  help  ensure  the 
alternatives  to  be  analyzed  are  sufficient,  diverse,  and  viable. 

3.  Analyze  alternatives:  includes  best  practices  that  compare  the 
alternatives  to  be  analyzed.  The  best  practices  in  this  category  help 
ensure  that  the  team  conducting  the  analysis  uses  a  standard, 
quantitative  process  to  assess  the  alternatives. 

4.  Document  and  review  the  AOA  process:  includes  best  practices 
that  would  be  applied  throughout  the  AOA  process,  such  as 
documenting  all  steps  taken  to  initialize,  identify,  and  analyze 
alternatives  and  to  select  a  preferred  alternative  in  a  single  document. 


^he  best  practices  listed  in  this  appendix  are  an  update  of  and  supersede  the  initial  set  of 
24  best  practices  listed  in  GAO-15-37. 
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5.  Select  a  preferred  alternative:  includes  a  best  practice  that  is 

applied  by  the  decision  maker  to  compare  alternatives  and  to  select  a 
preferred  alternative. 

The  five  phases  address  different  themes  of  analysis  necessary  to 
complete  the  AOA  process  and  comprise  the  beginning  of  the  AOA 
process  (defining  the  mission  needs  and  functional  requirements)  through 
the  final  step  of  the  AOA  process  (select  a  preferred  alternative). 

There  are  three  key  entities  that  are  involved  in  the  AOA  process:  the 
customer,  the  decision  maker,  and  the  AOA  team.  The  customer  refers  to 
the  program  office,  service,  or  agency  that  identifies  a  mission  need  (e.g. 
a  credible  gap  between  current  capabilities  and  those  required  to  meet 
the  goals  articulated  in  the  strategic  plan).  The  decision  maker  is  the 
person  or  entity  that  signs  off  on  the  final  decision  and  analysis 
documented  by  the  AOA  report.  The  decision  maker  refers  to  the  program 
manager  (or  alternate  authority  figure  identified  early  in  the  AOA  process) 
who  will  select  the  preferred  alternative  based  on  the  established 
selection  criteria.  The  AOA  team  is  the  group  of  subject  matter  experts 
who  are  involved  in  the  day-to-day  work  of  the  AOA  process  and  work  to 
develop  the  analysis  that  is  the  foundation  of  the  AOA  process. 

Conforming  to  the  22  best  practices  helps  ensure  that  the  preferred 
alternative  selected  is  the  one  that  best  meets  the  agency’s  mission 
needs.  Not  conforming  to  the  best  practices  may  lead  to  an  unreliable 
AOA,  and  the  customer  will  not  have  assurance  that  the  preferred 
alternative  best  meets  the  mission  needs.  Table  1  shows  the  22  best 
practices  and  the  five  phases. 
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Table  1:  Best  Practices  for  the  Analysis  of  Alternatives  (AOA)  Process 


Phase  I.  Initialize  the  AOA  process 

1.  Define  mission  need 

Definition :  The  customer  defines  the  mission  needs  (i.e.,  a  credible  gap  between  current  capabilities  and  those  required  to  meet  the 
goals  articulated  in  the  strategic  plan)  without  a  predetermined  solution.  To  ensure  that  the  AOA  process  does  not  favor  one  solution 
over  another,  the  AOA  is  conducted  before  design  and  development  of  the  required  capabilities.  The  customer  decides  at  which  level 
of  design  completion  an  AOA  should  be  performed,  with  the  understanding  that  the  more  complete  the  design,  the  more  information  is 
available  to  support  a  robust  analysis  and  to  select  a  preferred  alternative  that  best  meets  the  mission  need. 

Effect  Allowing  mission  needs  to  be  defined  in  solution-specific  terms  creates  a  potential  bias  and  could  invalidate  the  analysis. 

2.  Define  functional  requirements 

Definition:  The  customer  defines  functional  requirements  (i.e.  the  general  parameters  that  the  selected  alternative  must  have  to 
address  the  mission  need)  based  on  the  mission  need  without  a  predetermined  solution.  The  customer  defines  the  capabilities  that 
the  AOA  process  seeks  to  refine  through  characterized  gaps  between  capabilities  in  the  current  environment  and  the  capabilities 
required  to  meet  the  stated  objectives  for  the  future  environment.  These  functional  requirements  are  realistic,  organized,  clear, 
prioritized,  and  traceable.  It  is  advisable  that  functional  requirements  be  set  early  in  the  AOA  process  and  agreed  upon  by  all 
stakeholders. 

Effect:  The  AOA  process  is  tied  to  the  identified  mission  needs.  Setting  functional  requirements  to  a  standard  other  than  mission 
needs  allows  bias  to  enter  the  study  because  the  requirements  might  then  reflect  arbitrary  measures.  Additionally,  requirements  not 
tied  to  mission  needs  make  it  difficult  to  quantify  the  benefits  of  each  alternative  relative  to  what  is  required  and  make  it  challenging  for 
decision  makers  to  assess  which  capability  gaps  will  be  met  for  each  alternative. 

3.  Develop  AOA  time  frame 

Definition:  The  customer  provides  the  team  conducting  the  analysis  enough  time  to  complete  the  AOA  in  order  to  ensure  a  robust  and 
complete  analysis.  Since  an  AOA  process  requires  a  large  team  with  many  diverse  resources  and  expertise,  the  process  requires 
sufficient  time  to  be  accomplished  thoroughly.  A  detailed  schedule  is  developed  prior  to  starting  the  AOA  process.  The  duration  of  the 
AOA  process  depends  on  the  number  of  viable  alternatives  and  availability  of  the  team  members.  The  time  frame  is  tailored  for  the 
type  of  system  to  be  analyzed  and  ensures  that  there  is  adequate  time  to  accomplish  all  of  the  AOA  process  steps  robustly. 

Effect:  The  AOA  process  identifies  and  thoroughly  analyzes  a  comprehensive  range  of  alternatives.  Recommending  an  alternative 
without  adequate  time  to  perform  the  analysis  is  a  contributing  factor  to  high  dollar  acquisitions  that  have  significantly  overrun  both 
cost  and  schedule  while  falling  short  of  expected  performance. 

4.  Establish  AOA  team 

Definition:  After  the  customer  establishes  the  need  for  the  AOA  in  steps  1  through  3,  a  diverse  AOA  team  is  established  to  develop  the 
AOA.  This  team  consists  of  members  with  a  variety  of  necessary  skill  sets,  specific  knowledge,  and  abilities  to  successfully  execute 
the  study.  For  example,  the  AOA  team  includes  individuals  with  skills  and  experience  in  the  following  areas:  program  management, 
federal  contracting,  cost  estimating,  risk  management,  sustainability,  scheduling,  operations,  technology,  earned  value  management, 
budget  analysis,  and  any  other  necessary  areas  of  expertise. 

Effect:  An  AOA  process  includes  a  diverse  group  of  subject  matter  experts  (SMEs)  to  perform  the  analysis.  Since  each  SME  brings 
their  knowledge  to  the  team,  without  the  appropriate  expertise  on  the  team,  errors  in  the  results  could  occur  and  gaps  in  the  analysis 
could  be  created,  causing  the  AOA’s  completion  to  be  delayed  as  more  SMEs  are  identified  and  tasked  to  work  as  part  of  the  AOA 
process. 

5.  Define  selection  criteria 

Definition:  The  AOA  team  or  the  decision  maker  defines  selection  criteria  based  on  the  mission  need.  The  defined  criteria  are  based 
on  mission  needs  and  are  independent  of  a  particular  capital  asset  or  technological  solution.  The  selection  criteria  are  defined  based 
on  the  mission  need  prior  to  starting  the  analysis. 

Effect:  It  is  essential  that  the  selection  criteria  be  based  on  the  mission  needs.  If  there  are  no  preset  criteria  based  on  documented 
requirements,  bias  can  enter  the  AOA  process  and  prevent  the  decision  maker  from  forming  an  impartial  and  unbiased  decision. 
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6.  Weight  selection  criteria 

Definition:  The  AOA  team  or  the  decision  maker  weights  the  selection  criteria  to  reflect  the  relative  importance  of  each  criterion.  While 
the  selection  criteria  are  ranked  in  importance,  the  alternatives  are  based  on  trade-offs  between  costs,  operational  effectiveness, 
risks,  schedules,  flexibility,  and  other  factors  identified  by  the  team  or  the  decision  maker. 

Effect:  An  unjustified  weighting  method  can  oversimplify  the  results  and  potentially  mask  important  information  leading  to  an 
uninformed  decision. 

7.  Develop  AOA  process  plan 

Definition:  The  AOA  team  creates  a  plan  to  include  proposed  methodologies  for  identifying,  analyzing,  and  selecting  alternatives  prior 
to  beginning  the  AOA  process.  This  plan  establishes  the  critical  questions  to  be  explored,  the  selection  criteria,  the  basis  of  estimates, 
and  measures  that  are  used  to  rate,  rank,  and  decide  among  the  alternatives.  Additionally,  the  plan  includes  the  criteria  used  to 
determine  each  alternative’s  viability.  A  road  map  and  standard  work  breakdown  structure  (WBS)  are  used  to  compare  the 
alternatives  with  the  baseline  and  with  each  other. 

Effect:  The  functional  requirements  and  selection  criteria  are  identified  prior  to  the  beginning  of  the  analysis.  If  criteria  to  select  the 
preferred  alternative  are  established  after  the  analysis  has  begun  bias  may  influence  the  study’s  results.  Furthermore,  if  planned 
methodologies  for  the  remaining  phases  of  the  AOA  study  are  not  established,  the  risk  of  applying  poor  methodologies  as  part  of  the 
AOA  analysis  increases. 

Phase  II.  Identify  alternatives 

8.  Develop  list  of  alternatives 

Definition:  The  AOA  team  identifies  and  considers  a  diverse  range  of  alternatives  to  meet  the  mission  need.  To  fully  address  the 
capability  gaps  between  the  current  environment  and  the  stated  objectives  for  the  future  environment,  market  surveillance  and  market 
research  is  performed  to  develop  as  many  alternative  solutions  as  possible  for  examination.  Alternatives  are  mutually  exclusive,  that 
is,  the  success  of  one  alternative  does  not  rely  upon  the  success  of  another. 

Effect:  An  AOA  process  encompasses  numerous  alternatives  in  order  to  ensure  that  the  study  provides  a  broad  view  of  the  issue.  If 
the  AOA  team  does  not  perform  thorough  research  to  capture  diverse  alternatives,  the  optimal  alternative  could  be  overlooked  and 
invalidate  the  AOA’s  results  and  bias  the  process. 

9.  Describe  alternatives 

Definition:  The  AOA  team  describes  alternatives  in  sufficient  detail  to  allow  for  robust  analysis.  All  alternatives’  scope  is  described  in 
terms  of  functional  requirements.  This  description  is  detailed  enough  to  support  the  viability,  cost,  and  benefit/effectiveness  analyses. 
Effect:  Documentation  is  essential  for  validating  the  AOA  process  and  defending  its  conclusions.  Unless  the  AOA  team  adequately 
describes  and  documents  the  alternatives,  the  analysis  will  not  provide  sufficient  detail  to  allow  for  valid  cost-benefit  estimates  and  will 
not  be  credible. 

10.  Include  baseline  alternative 

Definition:  The  AOA  team  includes  one  alternative  to  represent  the  status  quo  to  provide  a  basis  of  comparison  among  alternatives.  It 
is  critical  for  the  AOA  team  to  first  understand  the  status  quo,  which  represents  the  existing  capability’s  baseline  where  no  action  is 
taken,  before  comparing  alternatives.  The  baseline  is  well  documented  as  an  alternative  in  the  study  and  is  used  to  represent  the 
current  capabilities  and  also  for  explicit  comparison  later  in  the  study. 

Effect:  It  is  essential  that  the  AOA  process  compare  the  current  environment  with  the  possible  future  environment.  If  no  status  quo  is 
examined,  then  there  is  no  benchmark  for  comparison,  allowing  arbitrary  comparisons  between  alternatives  and  hindering  the 
credibility  of  the  study. 
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1 1 .  Assess  alternatives’  viability 

Definition:  The  AOA  team  screens  the  list  of  alternatives  to  eliminate  those  alternatives  that  are  not  viable,  and  it  documents  the 
reasons  for  eliminating  any  alternatives.  All  alternatives  are  examined  using  predetermined  qualitative  technical  and  operational 
factors  to  determine  their  viability.  Only  those  alternatives  found  viable  are  examined  fully  in  the  AOA  process.  However,  all 
assumptions  regarding  the  alternatives’  viable  and  nonviable  status  are  fully  documented,  including  reasons  that  an  alternative  is  not 
viable,  in  order  to  justify  the  recommendation.  Additionally,  viable  alternatives  that  are  not  affordable  within  the  projected  available 
budget  are  dropped  from  final  consideration. 

Effect:  Not  eliminating  alternatives  based  on  viability  could  needlessly  extend  the  study’s  duration  and  burden  the  AOA  team  or  lead  to 
the  selection  of  a  technically  nonviable  alternative.  Furthermore,  unless  the  AOA  team  considers  affordability  as  part  of  the  final 
recommendation,  an  alternative  that  is  not  feasible  based  on  the  current  fiscal  environment  could  be  selected.  Documenting  the 
alternatives  that  are  deemed  nonviable  is  important  so  that  decision  makers  can  clearly  see  why  those  alternatives  are  not  considered 
for  further  analysis. 

Phase  III.  Analyze  alternatives 

12.  Identify  significant  risks  and  mitigation  strategies 

Definition:  The  AOA  team  identifies  and  documents  the  significant  risks  and  mitigation  strategies  for  each  alternative.  Risks  are 
ranked  in  terms  of  significance  to  mission  needs  and  functional  requirements.  All  risks  are  documented  for  each  alternative  along  with 
any  overarching  or  alternative  specific  mitigation  strategies.  Schedule  risk,  cost  risk,  technical  feasibility,  risk  of  technical 
obsolescence,  dependencies  between  a  new  project  and  other  projects  or  systems,  procurement  and  contract  risk,  and  resources 
risks  are  examined. 

Effect:  Since  AOA  processes  typically  occur  early  in  the  acquisition  process,  risk  is  inherently  a  part  of  every  alternative.  Not 
documenting  the  risks  and  related  mitigation  strategies  for  each  alternative  prevents  decision  makers  from  performing  a  meaningful 
trade-off  analysis  necessary  to  choose  a  recommended  alternative. 

13.  Determine  and  quantify  benefits/effectiveness 

Definition:  The  AOA  team  uses  a  standard  process  to  document  the  benefits  and  effectiveness  of  each  alternative.  The  AOA  team 
drafts  a  metric  framework  that  details  the  methods  used  to  evaluate  and  quantify  the  measures  of  effectiveness  and  measures  of 
performance  for  all  mission  needs.  The  AOA  team  quantifies  the  benefits  and  effectiveness  of  each  alternative  over  the  alternative’s 
full  life-cycle,  if  possible.  Just  as  costs  cover  the  entire  life-cycle  for  each  alternative,  the  benefits  and  effectiveness  measures  cover 
each  alternative’s  life-cycle,  if  possible,  in  order  to  determine  each  alternative’s  net  present  value  (NPV) — the  discounted  value  of 
expected  benefits  minus  the  discounted  value  of  expected  costs.  In  cases  where  the  means  to  monetize  a  benefit  are  too  vague  (for 
example,  intangibles  like  scientific  knowledge),  the  AOA  team  treats  those  benefits  as  strategic  technical  benefits  and  uses  scalability 
assessments  to  quantify  those  benefits  so  that  they  are  compared  across  all  viable  alternatives.  In  situation  where  benefits  cannot  be 
quantified,  the  AOA  team  explains  why  this  is  the  case  as  part  of  their  analysis. 

Effect:  Determining  a  standard  process  to  quantify  benefits  is  an  essential  part  of  the  AOA  process.  If  the  AOA  team  does  not  clearly 
establish  criteria  against  which  to  measure  all  alternatives,  bias  is  introduced  to  the  study.  Additionally,  if  the  AOA  team  does  not 
examine  effectiveness  over  the  entire  life-cycle,  decision  makers  cannot  see  the  complete  picture  and  are  prevented  from  making  an 
informed  decision. 

14.  Tie  benefits/effectiveness  to  mission  need 

Definition:  The  AOA  team  explains  how  each  measure  of  effectiveness  supports  the  mission  need.  The  AOA  team  shows  how  the 
measures  of  effectiveness  describe  the  way  the  current  environment  is  expected  to  evolve  to  meet  the  desired  environment;  the  team 
also  shows  how  the  measures  are  tied  to  specific  mission  needs  and  functional  requirements.  This  is  the  hierarchy  that  connects  the 
overarching  requirements  to  the  data  that  are  needed. 

Effect:  Unless  the  AOA  team  thoroughly  documents  how  the  measures  of  effectiveness  relate  to  specific  mission  needs  and  functional 
requirements,  decision  makers  will  not  have  proper  insight  into  the  impact  of  each  alternative. 
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15.  Develop  life-cycle  cost  estimates  (LCCEs) 

Definition :  The  AOA  team  develops  a  LCCE  for  each  alternative,  including  all  costs  from  inception  of  the  project  through  design, 
development,  deployment,  operation,  maintenance,  and  disposal.  The  AOA  team  includes  a  cost  expert  who  is  responsible  for 
development  of  a  comprehensive,  well-documented,  accurate,  and  credible  cost  estimate  for  each  viable  alternative  in  the  study.  The 
LCCE  for  each  alternative  follows  the  GAO  12-step  guide  and  uses  a  common  cost  element  structure  for  all  alternatives  and  includes 
all  costs  for  each  alternative.3  Costs  that  are  the  same  across  the  alternatives  (for  example,  training  costs)  are  included  so  that 
decision  makers  can  compare  the  total  cost  rather  than  just  the  portion  of  costs  that  varies  across  all  viable  alternatives.  The  AOA 
team  expresses  the  LCCE  in  present  value  terms  and  explains  why  it  chose  the  specific  discount  rate  used.  The  AOA  team  ensures 
that  economic  changes,  such  as  inflation  and  the  discount  rate  are  properly  applied,  realistically  reflected,  and  documented  in  the 
LCCE  for  all  alternatives.  Furthermore,  the  present  value  of  the  estimate  reflects  the  time  value  of  money — the  concept  that  a  dollar 
today  can  be  invested  and  earn  interest. 

Effect  An  LCCE  that  is  incomplete  (i.e.  does  not  include  all  phases  of  an  alternative’s  life-cycle)  does  not  provide  an  accurate  and 
complete  view  of  the  alternatives’  costs.  Without  a  full  accounting  of  life-cycle  costs,  decision  makers  will  not  have  a  complete  picture 
of  the  costs  for  each  alternative  and  will  have  difficulty  comparing  the  alternatives  because  comparisons  may  not  be  based  on 
accurate  information.  Additionally,  applying  a  discount  rate  is  an  important  step  in  cost  estimating  because  all  cost  data  must  be 
expressed  in  like  terms  for  comparison.  Unless  the  AOA  team  properly  normalizes  costs  to  a  common  standard,  any  comparison 
would  not  be  accurate,  and  any  recommendations  resulting  from  the  flawed  analysis  would  be  negated.  Properly  normalizing  costs  is 
particularly  important  if  various  alternatives  have  different  life-cycles. 

16.  Include  a  confidence  interval  or  range  for  LCCEs 

Definition:  The  AOA  team  presents  the  LCCE  for  each  alternative  with  a  confidence  interval  or  range,  and  not  solely  as  a  point 
estimate.  To  document  the  level  of  risk  associated  with  the  point  estimate  for  each  viable  alternative,  the  confidence  interval  is 
included  as  part  of  the  LCCE  for  each  viable  alternative  (in  accordance  with  GAO  Cost  Estimating  Best  Practice  #9,  risk  and 
uncertainty  analysis).15  Decision  makers  must  have  access  to  the  confidence  interval  associated  with  the  point  estimates  for  all  viable 
alternatives  in  order  to  make  informed  decisions.  Additionally,  the  AOA  team  uses  a  consistent  method  of  comparing  alternatives  in 
order  to  present  a  comparable  view  of  the  risk  associated  with  each  alternative.  For  example,  the  comparison  can  be  based  on  an 
established  dollar  value  across  alternatives  (in  order  to  observe  the  confidence  level  for  each  alternative  at  that  dollar  value). 
Alternatively,  the  comparison  can  be  based  on  a  predetermined  confidence  level  across  alternatives  (in  order  to  observe  the  dollar 
value  associated  with  that  confidence  level  for  each  alternative). 

Effect:  For  decision  makers  to  make  an  informed  decision,  the  alternatives’  LCCEs  must  reflect  the  degree  of  uncertainty.  Having  a 
range  of  costs  around  a  point  estimate  is  useful  because  it  conveys  a  level  of  confidence  for  each  alternative  to  achieve  a  most  likely 
cost.  Without  cost  risk  and  uncertainty  analysis  the  LCCEs  for  the  viable  alternatives  are  not  credible. 

17.  Perform  sensitivity  analysis 

Definition:  The  AOA  team  tests  and  documents  the  sensitivity  of  the  cost  and  benefit  and  effectiveness  estimates  for  each  alternative 
to  risks  and  changes  in  key  assumptions.  Major  outcomes  and  assumptions  are  varied  in  order  to  determine  each  alternative’s 
sensitivity  to  changes  in  key  assumptions.  This  analysis  is  performed  in  order  to  rank  the  key  drivers  that  could  influence  the  cost  and 
benefit  estimates  based  on  how  they  affect  the  final  results  for  each  alternative.  Each  alternative  includes  both  a  sensitivity  and  risk 
and  uncertainty  analysis  that  identifies  a  range  of  possible  costs  based  on  varying  key  assumptions,  parameters,  and  data  inputs.  As 
explained  in  best  practice  #16,  life-cycle  cost  estimates  are  adjusted  to  account  for  risk  and  sensitivity  analyses. 

Effect:  Failing  to  conduct  a  sensitivity  analysis  to  identify  the  uncertainties  associated  with  different  assumptions  increases  the  chance 
the  AOA  team  will  recommend  an  alternative  without  an  understanding  of  the  full  impacts  on  life-cycle  costs,  which  could  lead  to  cost 
and  schedule  overruns. 


Page  29 


GAO-16-22  Amphibious  Combat  Vehicle 


Appendix  I:  Best  Practices  for  the  Analysis  of 
Alternatives  Process 


Phase  IV.  Document  and  review  the  AOA  process 

18.  Document  AOA  process  in  a  single  document 

Definition:  The  AOA  team  documents  all  steps  taken  to  identify,  analyze,  and  select  alternatives  in  a  single  document.  This  document 
clearly  states  the  preferred  alternative  and  provides  the  detailed  rationale  for  the  recommendation  based  on  analytic  results.  The 
report  includes  sections  detailing  the  steps  taken  to  initialize  the  AOA  process,  and  to  identify,  analyze,  and  select  alternatives.  For 
example,  one  section  lists  the  overall  selection  criteria  and  rationale  for  nonviable  or  viable  ratings  for  alternatives,  assumptions  for 
each  alternative,  risk  drivers  and  mitigation  techniques,  analysis  of  the  costs  and  benefits  associated  with  each  alternative,  and  the 
trade-offs  between  costs,  benefits,  and  risks. 

Effect:  Documentation  is  essential  for  validating  and  defending  the  AOA  process.  Without  clear  reports  that  compile  all  information, 
including  standards  used  to  rate  and  perform  the  analysis,  the  study’s  credibility  could  suffer  because  the  documentation  does  not 
explain  the  rationale  for  methodology  or  the  calculations  underlying  the  analysis.  Having  all  the  information  related  to  all  best  practices 
of  the  AOA  process  in  one  single  document  also  makes  it  easier  for  an  independent  reviewer  to  assess  the  AOA  process. 

19.  Document  assumptions  and  constraints 

Definition:  The  AOA  team  documents  and  justifies  all  assumptions  and  constraints  used  in  the  AOA  process.  Assumptions  and 
constraints  help  to  scope  the  AOA.  Assumptions  are  explicit  statements  used  to  specify  precisely  the  environment  to  which  the 
analysis  applies,  while  constraints  are  requirements  or  other  factors  that  cannot  be  changed  to  achieve  a  more  beneficial  approach. 
Both  assumptions  and  constraints  are  detailed  and  justified  for  each  alternative  in  the  AOA  plan. 

Effect:  Without  documented  and  justified  assumptions  and  constraints  it  will  be  difficult  for  decision  makers  to  evaluate  between  the 
alternatives. 

20.  Ensure  AOA  process  is  impartial 

Definition:  The  AOA  team  conducts  the  analysis  without  a  predetermined  solution.  The  AOA  process  informs  the  decision-making 
process  rather  than  reflecting  the  validation  of  a  predetermined  solution.  The  AOA  process  is  an  unbiased  inquiry  into  the  costs, 
benefits,  and  capabilities  of  all  alternatives. 

Effect:  An  AOA  process  is  not  considered  valid  if  it  is  biased.  Performing  a  study  with  a  predetermined  solution  distorts  the  results. 

The  validity  of  the  analysis  is  affected  if  bias  is  introduced  to  the  inputs. 

21.  Perform  independent  review 

Definition:  An  entity  independent  of  the  AOA  process  reviews  the  extent  to  which  all  best  practices  are  followed.  The  AOA  process  is 
completed  with  enough  thoroughness  to  ensure  that  an  independent  organization  outside  of  the  project’s  chain  of  command  can 
review  the  AOA  documentation  and  clearly  understand  the  process  and  rationale  that  led  to  the  selection  of  the  recommended 
alternative.  Part  of  the  documentation  includes  approval  and  review  from  an  office  outside  of  the  one  that  asked  for  or  performed  the 
AOA  process.  For  certain  projects,  in  addition  to  an  independent  review  at  the  end  of  the  AOA  process,  additional  independent 
reviews  are  necessary  at  earlier  stages  of  the  process,  such  as  reviews  of  the  AOA  process  plan  of  the  identification  of  viable 
alternatives.  While  early  reviews  are  not  a  substitute  for  the  independent  review  conducted  at  the  end  of  the  AOA  process,  they  help 
ensure  that  bias  is  not  added  throughout  the  course  of  the  AOA  process. 

Effect:  An  independent  review  is  one  of  the  most  reliable  means  to  validate  an  AOA  process.  Without  an  independent  review,  the 
results  are  more  likely  to  include  organizational  bias  or  lack  the  thoroughness  needed  to  ensure  that  a  preferred  solution  is  chosen 
and  not  a  favored  solution. 

Phase  V.  Select  a  preferred  alternative 

22.  Compare  alternatives 

Definition:  The  AOA  team  or  the  decision  maker  compares  the  alternatives  using  NPV,  if  possible,  to  select  a  preferred  alternative. 
NPV  can  be  negative  if  discounted  costs  are  greater  than  discounted  benefits.  NPV  is  the  standard  criteria  used  when  deciding 
whether  an  alternative  can  be  justified  based  on  economic  principles.  In  some  cases,  NPV  cannot  be  used,  such  as  when  quantifying 
benefits  is  not  possible.  In  these  cases,  the  AOA  team  documents  why  NPV  cannot  be  used.  Furthermore,  if  NPV  is  not  used  to 
differentiate  among  alternatives,  the  AOA  team  should  document  why  NPV  is  not  used,  and  describe  the  other  method  that  is  used  to 
differentiate,  and  explain  why  that  method  has  been  applied. 

Effect:  Comparing  items  that  have  not  been  discounted  (or  normalized)  does  not  allow  for  time  series  comparisons  since  alternatives 
may  have  different  life  cycles  or  different  costs  and  benefits. 


Source:  GAO.  |  GAO-16-22 
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aGAO,  GAO  Cost  Estimating  and  Assessment  Guide:  Best  Practices  for  Developing  and  Managing 
Capital  Program  Costs,  GAO-09-3SP  (Washington,  D.C.:  Mar.  2,  2009). 

bGAO-09-3SP. 

Some  best  practices  included  in  a  phase  can  take  place  concurrently  and 
do  not  have  to  follow  the  order  presented  in  table  1 .  The  phases  should 
occur  in  sequence  to  prevent  bias  from  entering  the  analysis  and  adding 
risk  that  the  AOA  team  will  analyze  alternatives  that  have  not  been 
defined.  However,  the  document  and  review  phase  can  be  done  at  any 
stage  throughout  the  AOA  process.  For  example,  best  practice  5  (define 
selection  criteria)  can  be  done  at  the  same  time  as  best  practice  6  (weight 
selection  criteria).  On  the  other  hand,  best  practice  20  (ensure  AOA 
process  is  impartial)  can  be  done  at  the  end  of  every  step  or  every  phase 
to  ensure  the  impartiality  of  the  AOA  as  it  progresses.  The  best  practices 
represent  an  overall  process  that  results  in  a  reliable  AOA  that  can  be 
easily  and  clearly  traced,  replicated,  and  updated.  Figure  5  shows  the 
AOA  process  and  how  the  steps  in  each  phase  are  interrelated. 
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Figure  5:  AOA  Process  Chart 


Document  and  review 


AOA  Analysis  of  alternatives 
LCCE  Life-cycle  cost  estimate 

Source:  GAO.  |  GAO-16-22 


Note:  The  figure  displays  the  AOA  process  by  phase  and  step.  The  “Initialize,  Identify,  Analyze,  and 
Select”  phases  should  be  done  in  order,  but  the  “Document  and  review”  phase  can  be  done 
throughout  the  AOA  process.  The  arrows  indicate  that  the  “Document  and  review”  phase  is  related  to 
the  other  four  phases.  Within  each  phase,  there  are  steps  that  can  be  done  concurrently  rather  than 
consecutively.  The  concurrent  steps  are  grouped  together  in  dark  blue  boxes.  Furthermore,  there  are 
steps  in  later  phases  that  are  related  to  steps  in  earlier  phases;  these  are  connected  with  a  two  way 
arrow 


Review  the  AOA 
Process 


An  important  best  practice  is  an  independent  review  of  the  AOA  process. 
It  is  important  that  the  AOA  process  and  its  results  be  validated  by  an 
organization  independent  of  the  program  office  and  the  project’s  chain  of 
command,  to  ensure  that  a  high-quality  AOA  is  developed,  presented, 
and  defended  to  management.  This  process  verifies  that  the  AOA 
adequately  reflects  the  program’s  mission  needs  and  provides  a 
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reasonable  assessment  of  the  cost  and  benefits  associated  with  the 
alternatives. 

One  reason  to  independently  validate  the  AOA  process  is  that 
independent  reviewers  typically  rely  less  on  assumptions  alone  and, 
therefore,  tend  to  provide  more  realistic  analyses.  Moreover,  independent 
reviewers  are  less  likely  to  automatically  accept  unproven  assumptions 
associated  with  anticipated  savings.  That  is,  they  bring  more  objectivity  to 
their  analyses,  resulting  in  a  reality  check  of  the  AOA  process  that 
reduces  the  odds  that  management  will  invest  in  an  unreasonable 
alternative. 

To  that  end,  we  established  four  characteristics  that  identify  a  high- 
quality,  reliable  AOA  process.  These  characteristics  would  evaluate  if  the 
AOA  process  is  well-documented,  comprehensive,  unbiased,  and 
credible. 

•  “Well-documented”  means  that  the  AOA  process  is  thoroughly 
described  in  a  single  document,  including  all  source  data,  clearly 
detailed  methodologies,  calculations  and  results,  and  that  selection 
criteria  are  explained. 

•  “Comprehensive”  means  that  the  AOA  process  ensures  that  the 
mission  need  is  defined  in  a  way  to  allow  for  a  robust  set  of 
alternatives,  that  no  alternatives  are  omitted  and  that  each  alternative 
is  examined  thoroughly  for  the  project’s  entire  life-cycle. 

•  “Unbiased”  means  that  the  AOA  process  does  not  have  a 
predisposition  toward  one  alternative  over  another;  it  is  based  on 
traceable  and  verifiable  information. 

•  “Credible”  means  that  the  AOA  process  thoroughly  discusses  the 
limitations  of  the  analyses  resulting  from  the  uncertainty  that 
surrounds  both  the  data  and  the  assumptions  for  each  alternative. 

Table  2  shows  the  four  characteristics  and  their  relevant  AOA  best 
practices. 
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Table  2:  Criteria  and  Characteristics 


Characteristics  AOA  process  best  practice 

Well  documented:  The  analysis  of  alternatives  (AOA)  process  is  thoroughly  12.  Identify  significant  risks  and  mitigation  strategies 

described,  including  all  source  data,  clearly  detailed  methodologies, 

calculations  and  results,  and  selection  criteria  are  explained. 

•  Includes  a  detailed  list  of  ground  rules,  assumptions,  risks,  and 
mitigation  strategies  needed  to  provide  a  robust  analysis  for  all 
alternatives. 

•  Explains  how  each  alternative’s  identified  measures  of 
benefits/effectiveness  support  the  mission  needs. 

.  Details  in  a  single  document  all  processes,  criteria,  and  data  used  to 
support  the  AOA  process’s  final  decision. 

•  Describes  the  estimating  methodology  and  rationale  used  to  build  costs 
and  benefits  for  all  alternatives. 


14.  Tie  benefits/effectiveness  to  mission  need 

18.  Document  AOA  process  in  a  single  document 

19.  Document  assumptions  and  constraints 


Comprehensive:  The  level  of  detail  for  the  AOA  process  ensures  no 
alternatives  are  omitted  and  that  each  alternative  is  examined  thoroughly  for 
the  project’s  entire  life-cycle. 

•  Identifies  and  screens  a  diverse  range  of  alternatives. 

•  Compares  alternatives  across  their  entire  life-cycle  rather  than  focusing 
on  one  phase  of  the  acquisition  process. 

Unbiased:  The  AOA  process  does  not  have  a  predisposition  towards  one 
alternative  over  another  but  is  based  on  traceable  and  verified  information 

•  Defines  the  mission  needs  and  functional  requirements  independently 
of  an  operational  solution. 

.  Ensures  that  the  appropriate  personnel  are  assigned  to  the  task  and 
there  is  enough  time  to  complete  a  thorough  study. 

•  Documents  a  standard  process  that  defines  selection  criteria  based  on 
mission  need  and  quantifies  the  benefit/effectiveness  measures  to 
ensure  the  AOA  process  is  conducted  without  a  pre-determined  solution 
in  mind. 


1.  Define  mission  need 

3.  Develop  AOA  timeframe 
8.  Develop  list  of  alternatives 

1 1 .  Assess  alternatives’  viability 

15.  Develop  life-cycle  cost  estimates  (LCCEs) 

2.  Define  functional  requirements 

4.  Establish  AOA  team 

6.  Weight  selection  criteria 

7.  Develop  AOA  process  plan 

13.  Determine  and  quantify  benefits  and  effectiveness 
20.  Ensure  AOA  process  is  impartial 
22.  Compare  alternatives 


.  Compares  solutions  based  on  pre-established  weighted  selection 
criteria  and  net  present  value  techniques. 


Credible:  The  AOA  process  discusses  any  limitations  of  the  analysis 
resulting  from  the  uncertainty  surrounding  the  data  to  assumptions  made  for 
each  alternative. 

•  Includes  a  baseline  scenario  as  the  benchmark  to  enable  comparison 
between  alternatives. 

.  Life-cycle  cost  estimates  developed  for  each  alternative  include  a 
confidence  interval  or  range  developed  based  on  risk/uncertainty 
analysis. 

•  Details  the  sensitivity  of  both  costs  and  benefits  to  changes  in  key 
assumptions  for  all  alternatives. 

•  Independent  review  of  the  AOA  process  is  performed  to  ensure  that  the 
study’s  results  are  logical  and  based  on  the  documented  data, 
assumptions,  and  analyses. 


5.  Define  selection  criteria 

9.  Describe  alternatives 

10.  Include  baseline  alternative 

16.  Include  a  confidence  interval  or  range  for  LCCEs 

17.  Perform  sensitivity  analysis 
21.  Perform  independent  review 


Source:  GAO.  |  GAO-16-22 
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To  determine  how  the  ACV  program’s  efforts  compare  with  best 
practices,  we  reviewed  program  documentation  and  other  materials  for 
the  ACV  acquisition,  including  the  acquisition  strategy,  technology 
readiness  assessment,  and  the  Capabilities  Development  Document.  We 
identified  acquisition  best  practices  based  on  our  extensive  body  of  work 
in  that  area  and  Department  of  Defense  (DOD)  guidance,  and  used  this 
information  to  analyze  the  proposed  ACV  acquisition  approach  and 
acquisition  activities  to  date.  We  also  reviewed  our  previous  work  on  the 
ACV  and  EFV  programs.  In  addition,  we  interviewed  program  and  agency 
officials  from  the  USMC’s  Advanced  Amphibious  Assault  program  office 
and  Combat  Development  and  Integration,  Analysis  Directorate,  the 
Office  of  the  Assistant  Secretary  of  the  Navy  for  Research,  Development, 
and  Acquisition,  and  the  Office  of  the  Secretary  of  Defense,  Cost 
Assessment  and  Program  Evaluation. 

To  determine  the  extent  to  which  the  2014  ACV  Analysis  of  Alternatives 
(AOA)  demonstrated  the  use  of  best  practices,  we  worked  with  USMC 
officials  to  identify  the  body  of  analyses  that  informed  the  2014  AOA. 
Different  pieces  of  each  report  or  analysis  in  the  full  body  of  work  were 
relevant  to  different  best  practices.  Because  the  2014  ACV  AOA  is  part  of 
a  larger  body  of  related  work  that  informs  this  analysis,  we  then  worked 
with  GAO  specialists  to  discuss  the  22  AOA  best  practices  and  categorize 
each  as  either  “individual”  or  “combined.”  Best  practices  labeled 
“individual”  have  been  assessed  based  on  only  the  2014  ACV  Analysis  of 
Alternatives  final  report.  Best  practices  noted  as  “combined”  were 
assessed  referring  to  the  full  body  of  work  that,  according  to  USMC 
officials,  has  informed  the  analysis  of  alternatives  process.  We  then 
compared  the  22  best  practices  to  the  2014  AOA  or  the  full  body  of  AOA 
analysis,  as  determined  above.  We  used  a  five-point  scoring  system  to 
determine  the  extent  to  which  the  AOA  conforms  to  best  practices.  To 
score  each  AOA  process,  (1 )  two  GAO  analysts  separately  examined  the 
AOA  documentation  received  from  the  agency  and  then  agreed  on  a 
score  for  each  of  the  22  best  practices,  then  (2)  a  GAO  AOA  specialist 
independent  of  the  engagement  team  reviewed  the  AOA  documentation 
and  the  scores  assigned  by  the  analysts  for  accuracy  and  cross-checked 
the  scores  in  all  the  analyses  for  consistency.  We  first  used  this  scoring 
system  to  determine  how  well  the  AOA  conformed  to  each  best  practice. 
We  then  used  the  average  of  the  scores  for  the  best  practices  in  each  of 
four  characteristics — well-documented,  comprehensive,  unbiased,  and 
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credible1 — to  determine  an  overall  score  for  each  characteristic.  We  sent 
our  draft  analysis  to  DOD  for  review.  They  provided  technical  comments 
and  additional  documentation  that  we  incorporated  to  ensure  our  analysis 
included  all  available  information.  We  then  used  the  same  methodology 
and  scoring  process  explained  above  to  revise  the  analysis  based  on 
their  technical  comments  and  any  additional  evidence  received.  If  the 
average  score  for  each  characteristic  was  “met”  or  “substantially  met,”  we 
concluded  that  the  AOA  process  conformed  to  best  practices  and 
therefore  could  be  considered  reliable.2 

To  determine  how  the  increments  of  ACV  are  to  achieve  amphibious 
capability,  we  reviewed  program  documentation  from  the  ACV  acquisition, 
including  the  acquisition  strategy  and  the  Concept  of  Employment,  as  well 
as  program  documentation  for  Navy  surface  connector  programs, 
including  the  Ship  to  Shore  Connector  Capabilities  Development 
Document  and  the  Surface  Connector  Council  charter.  We  also 
interviewed  USMC  officials  from  the  Combat  Development  and 
Integration,  Capabilities  Development  Directorate  and  Seabasing 
Integration  Division,  as  well  as  U.S.  Navy  officials  from  the  Naval  Sea 
Systems  Command. 

To  update  and  refine  the  AOA  best  practices  identified  in  prior  GAO  work, 
we  solicited  comments  from  a  set  of  over  900  internal  and  external 
experts  on  how  to  improve  the  previous  set  of  best  practices.  All 
comments  and  changes  were  vetted  during  three  vetting  sessions  with 
internal  GAO  experts.  The  resulting  changes  include  the  consolidation  of 
some  best  practices,  reducing  the  number  from  24  to  22,  and  the 
establishment  of  four  characteristics  that  identify  a  high-quality,  reliable 
AOA  process. 


Characteristics  are  defined  in  appendix  I. 

2The  five-point  scoring  system  that  we  used  was  as  follows:  “met”  means  that  USMC’s 
documentation  demonstrated  that  it  completely  met  the  best  practice;  “substantially  met” 
means  that  USMC’s  documentation  demonstrated  that  it  met  a  large  portion  of  the  best 
practice;  “partially  met”  means  that  USMC’s  documentation  demonstrated  that  it  met  about 
half  of  the  best  practice;  “minimally  met”  means  that  the  USMC’s  documentation 
demonstrated  that  it  met  a  small  portion  of  the  of  the  best  practice;  and  “did  not  meet” 
means  that  USMC’s  documentation  did  not  demonstrate  that  it  met  the  best  practice. 
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Appendix  III:  Amphibious  Combat  Vehicle 
2014  Analysis  of  Alternatives  Compared  with 
Best  Practices 


Overall,  the  DOD’s  ACV  analysis  of  alternatives  (AOA)  met  the  best 
practices  we  identified.  Table  3  below  describes  our  analysis  of  DOD’s 
AOA  compared  with  best  practices. 


Table  3:  Amphibious  Combat  Vehicle  Analysis  of  Alternatives  (AOA)  Compared  with  Best  Practices 


Best  practice 
phase 

Best  practice 

Combined  or 
Individual3 

Assessment 

Initialize  the  AOA 

1.  Define  mission  need 

Combined 

Met 

process 

2.  Define  functional  requirements 

Combined 

Met 

3.  Develop  AOA  timeframe 

Individual 

Met 

4.  Establish  AOA  team 

Individual 

Met 

5.  Define  selection  criteria 

Individual 

Met 

6.  Weight  selection  criteria 

Combined 

Met 

7.  Develop  AOA  process  plan 

Combined 

Met 

Identify 

8.  Develop  list  of  alternatives 

Combined 

Met 

alternatives 

9.  Describe  alternatives 

Combined 

Met 

10.  Include  baseline  alternative 

Combined 

Met 

1 1 .  Assess  alternatives’  viability 

Combined 

Met 

Analyze 

alternatives 

12.  Identify  significant  risks  and 
risk  mitigation  strategies 

Combined 

Met 

13.  Determine  and  quantify 
b  e  n  ef  i  ts/effe  ct  i  ve  n  ess 

Individual 

Substantially  met:  In  some  cases,  comparable 
mobility  analysis  data  was  not  available  for  all 
alternatives.  In  addition,  the  study  encountered  data 
consistency  issues  that  were  not  resolved  until  after 
the  analysis  of  alternatives  was  completed. 

14.  Tie  benefits/effectiveness  to 
mission  need 

Combined 

Met 

15.  Develop  life-cycle  cost 
estimates  (LCCEs) 

Individual 

Partially  met:  The  AOA  team  relied  on  cost 
estimating  experts  to  develop  a  reliable  life-cycle  cost 
estimate  for  each  alternative;  however,  while  the  cost 

and  affordability  analyses  use  inflation  indexes  to 
normalize  data,  they  do  not  adjust  for  a  nominal 
discount  rate  to  arrive  at  a  present  value  in  order  to 
account  for  both  potential  decreases  in  purchasing 
power  and  the  impact  of  the  future  value  of  money. 
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Appendix  III:  Amphibious  Combat  Vehicle  2014 
Analysis  of  Alternatives  Compared  with  Best 
Practices 


Best  practice 
phase 

Best  practice 

Combined  or 
Individual3 

Assessment13 

16.  Include  a  confidence  interval 
or  range  for  LCCEs 

Individual 

Substantially  met:  While  the  2014  AOA  study  plan 
presents  the  overall  cost  for  each  alternative  with  a 
low,  most  likely,  and  high  value;  none  of  these 
analyses  display  a  point  estimate  with  a  quantifiable 
confidence  level  or  associated  S-curve. 

17.  Perform  sensitivity  analysis 

Individual0 

Substantially  met:  Sensitivity  analysis  was  included  in 
the  cost  analysis,  affordability  assessment,  and  some 
elements  of  the  capability  assessments,  but  it  was 
not  performed  for  all  key  assumptions  in  each  case. 

Document  and 
review  the  AOA 
process 

18.  Document  AOA  process  in  a 
single  document 

Individual 

Substantially  Met:  The  2014  AOA  documented  all 
steps  taken  for  the  2014  analysis  and  makes 
reference  to  previous  studies  that  contributed  the 
analysis,  but  the  full  relationship  among  the  analyses 
is  not  specified  in  the  2014  report. 

19.  Document  assumptions  and 
constraints 

Combined 

Partially  met:  The  2014  analysis  includes  selected 
documentation  of  assumptions,  but  does  not  include 
a  full  list  of  assumptions,  constraints,  and 
justifications.  Any  relevant  assumptions  from 
previous  analyses  were  not  referenced  in  the  2014 
report. 

20.  Ensure  AOA  process  is 
impartial 

Combined 

Met 

21.  Perform  independent  review 

Individual 

Met 

Select  a  preferred 
alternative 

22.  Compare  alternatives 

Individual 

Minimally  met:  While  net  present  value  was  not  used 
in  the  report,  agency  officials  stated  that  they 
believed  the  cash  flows  and  discount  rates  were  such 
that  the  impact  was  minimal. 

Source:  GAO  analysis  of  USMC  information.  |  GAO-16-22 


aBest  practices  labeled  “Individual”  have  been  assessed  based  on  only  the  2014  ACV  Analysis  of 
Alternatives  final  report.  Best  practices  noted  as  “Combined”  were  assessed  referring  to  the  full  body 
of  work  that  has  informed  the  analysis  of  alternatives  process,  as  defined  by  USMC  officials.  This 
body  of  work  includes  studies  related  to  the  EFV  and  Advanced  Amphibious  Assault  Vehicle,  Marine 
Personnel  Carrier  and  ACV  requirements  documents,  the  2008  Marine  Personnel  Carrier  AOA,  the 
2012  ACV  AOA,  the  2014  ACV  AOA,  the  2013  ACV  Special  Project  on  high  water  speed,  and 
Systems  Engineering  Overarching  Product  Team  analysis. 

bThe  five-point  scoring  system  that  we  used  was  as  follows:  “met”  means  that  USMC’s  documentation 
demonstrated  that  it  completely  met  the  best  practice;  “substantially  met”  means  that  USMC’s 
documentation  demonstrated  that  it  met  a  large  portion  of  the  best  practice;  “partially  met”  means  that 
USMC’s  documentation  demonstrated  that  it  met  about  half  of  the  best  practice;  “minimally  met” 
means  that  the  USMC’s  documentation  demonstrated  that  it  met  a  small  portion  of  the  best  practice; 
and  “did  not  meet”  means  that  USMC’s  documentation  did  not  demonstrate  that  it  met  the  best 
practice. 

cAlthough  this  best  practice  would  usually  be  assessed  on  an  individual  basis,  the  2014  Amphibious 
Combat  Vehicle  AOA  report  specifically  refers  to  combat  scenarios  that  were  performed  for  the  ACV 
2012  AOA.  As  a  result,  GAO  considered  the  2012  scenario  information  to  be  subsumed  into  the  2014 
analysis  and  included  that  work  in  its  review  of  this  best  practice. 
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Appendix  III:  Amphibious  Combat  Vehicle  2014 
Analysis  of  Alternatives  Compared  with  Best 
Practices 


Table  4  provides  the  average  score  of  the  best  practices  under  each 
characteristic.  See  appendix  I  for  an  explanation  of  how  individual  best 
practices  are  grouped  under  each  characteristic.  Because  the  overall 
assessment  ratings  for  each  of  the  four  characteristics  are  substantially 
met  or  met,  we  concluded  that  the  AOA  process  conformed  to  best 
practices  and  can  be  considered  reliable. 


Table  4:  Average  Score  of  Best  Practices  for  each  Characteristic 

Characteristic 

Average  Score  of  Best  Practices 

Well-documented 

Substantially  met 

Comprehensive 

Met 

Unbiased 

Substantially  met 

Credible 

Met 

Source:  GAO  analysis  of  USMC  information.  |  GAO-16-22 
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Appendix  IV:  Additional  Information  on 
Selected  Surface  Connectors  and  Key 
Capabilities 


Surface  Connector  or 
Planned  Replacement 

Current 
Quantity  or 
Planned 
Quantity 

Water  Speed 
in  Knots3 

Payload  Capacity  in  Short 
Tons  (ST)  and  in  Number  of 
Amphibious  Combat  Vehicles 
(ACV)a,b 

Range  in 
Nautical 
Miles3 

Operating  Sea  State 
with  Significant  Wave 
Height  (SWH)C 

Landing  Craft  Air  Cushion 

72 

35 

60  ST 

100 

2  (1.0  SWH) 

Service  Life  Extension 

2  ACV 

Program 

Ship  to  Shore  Connector 

72 

35 

74  ST 

86 

3  (4. 1  SWH) 

2  ACV 

Landing  Craft  Utility 

32 

8 

125  STd 

1200 

3 

3  ACV 

Surface  Connector  (X) 

32 

8 

170  ST 

1200 

3 

Replacement 

3  ACV 

Expeditionary  Fast  Transport 

5 

35 

600  ST 

1200 

3 

11 

19  ACV 

Source:  GAO  presentation  of  US  Navy  information.  |  GAO-16-22 

aThe  figures  provided  for  each  capability  are  considered  maximums  for  current  surface  connectors  or 
expected  maximums  for  planned  replacement  surface  connectors.  Actual  performance  or  capabilities 
may  differ  due  to  environmental  or  mission-dependent  conditions,  such  as  payload  or  sea  state. 

bTransport  of  ACV  is  ship  to  shore. 

cLCAC  Capabilities  based  on  1 .0  significant  wave  height  (SWH).  Ship  to  Shore  Connector  capabilities 
based  on  4.1  SWH.  Both  the  North  Atlantic  Treaty  Organization  and  World  Meteorological 
Organization  sea  state  tables  characterize  1 .0  foot  SWH  as  sea  state  2  and  4.1  feet  SWH  as  sea 
state  4.  Sea  states  (SS)  are  classified  on  a  scale  of  0  to  9  depending  on  the  roughness  of  the  water 
as  caused  by  wind  or  other  disturbances.  SS  0  to  3  represent  calm  to  slight  seas  of  4  feet  or  less.  SS 
4  is  characterized  by  moderate  seas  of  4  to  8  feet.  SS  5  to  6  range  from  rough  to  very  rough  seas 
between  8  to  20  feet.  SS  7  to  9 — the  most  challenging  marine  conditions — reflect  high  to  extremely 
rough  seas,  including  seas  above  20  feet. 

dAII  LCU  were  de-rated  from  140  short  tons  due  to  corrosion  and  structural  issues  caused  by 
advanced  craft  age. 
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Appendix  V:  Comments  from  the  Department 
of  Defense 


PRINCIPAL  DEPUTY  UNDER  SECRETARY  OF  DEFENSE 
3015  DEFENSE  PENTAGON 
WASHINGTON.  DC  20301-3015 


ACQUISITION 
TECHNOLOGY, 
AND  LOGISTICS 


Ms.  Marie  A.  Mak 
Director 

Acquisition  and  Sourcing  Management 
U.S.  Government  Accountability  Office 
441  G  Street.  N.W. 

Washington,  DC  20548 


OCT  0  9  2015 


Dear  Ms.  Mak: 


This  is  the  Department  of  Defense  response  to  the  Government  Accountability  Office  (GAO) 
Draft  Report.  GAO-16-22,  “AMPHIBIOUS  COMBAT  VEHICLE  ACQUISITION:  Some 
Acquisition  Activities  Demonstrate  Best  Practices:  Attainment  of  Amphibious  Capability  to  be 
Determined."  dated  September  16,  2015  (GAO  Code  121275).  The  Department  appreciates  the 
effort  of  the  GAO  and  the  opportunity  to  comment  on  the  draft  report. 

The  report  appears  to  underestimate  ACV  I .  Us  planned  technical  maturity  and  associated 
risks.  The  proposed  vehicles  are  beyond  the  traditional  Preliminary  Design  Review  (PDR)  and 
Critical  Design  Review  (CDR)  level  of  maturity  due  to  vehicle  designs  based  on  fielded  systems 
that  are  being  modified  to  meet  Marine  Corps  requirement.  Conducting  a  combined  PDR/CDR 
is  appropriate  for  the  level  of  risk  identified  by  the  Program  Manager  and  will  establish  the  initial 
design  baseline  of  the  contractor  designs  that  w  ill  enter  into  developmental  and  operational  tests. 

We  believe  our  efforts  on  this  program  arc  aligned  with  GAO's  best  practices.  We  will 
continue  to  monitor  the  program  and  ensure  that  mitigations  are  in  place  to  address  potential  risk 
areas. 


My  point  of  contact  for  this  effort  is  Mr.  John  McGough  atjohn.t.mcgough.civ@mail.mil 
or  703-695-3043. 
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Appendix  VI:  GAO  Contact  and  Staff 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and 
policies;  and  provides  analyses,  recommendations,  and  other  assistance 
to  help  Congress  make  informed  oversight,  policy,  and  funding  decisions. 
GAO’s  commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no 
cost  is  through  GAO’s  website  (http://www.gao.gov).  Each  weekday 
afternoon,  GAO  posts  on  its  website  newly  released  reports,  testimony, 
and  correspondence.  To  have  GAO  e-mail  you  a  list  of  newly  posted 
products,  go  to  http://www.gao.gov  and  select  “E-mail  Updates.” 

Order  by  Phone 

The  price  of  each  GAO  publication  reflects  GAO’s  actual  cost  of 
production  and  distribution  and  depends  on  the  number  of  pages  in  the 
publication  and  whether  the  publication  is  printed  in  color  or  black  and 
white.  Pricing  and  ordering  information  is  posted  on  GAO’s  website, 
http://www.gao.gov/ordering.htm. 

Place  orders  by  calling  (202)  512-6000,  toll  free  (866)  801-7077,  or 

TDD  (202)  512-2537. 

Orders  may  be  paid  for  using  American  Express,  Discover  Card, 
MasterCard,  Visa,  check,  or  money  order.  Call  for  additional  information. 

Connect  with  GAO 

Connect  with  GAO  on  Facebook,  Flickr,  Twitter,  and  YouTube. 

Subscribe  to  our  RSS  Feeds  or  E-mail  Updates. 

Listen  to  our  Podcasts  and  read  The  Watchblog. 

Visit  GAO  on  the  web  at  www.gao.gov. 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Website:  http://www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Congressional 

Relations 

Katherine  Siggerud,  Managing  Director,  siggerudk@gao.gov,  (202)  512- 
4400,  U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room 

71 25,  Washington,  DC  20548 
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Chuck  Young,  Managing  Director,  youngd@gao.gov,  (202)512-4800 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7149 
Washington,  DC  20548 
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